DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE  • 


SYSTEMS  ACQUISITION  MANAGER'S  GUIDE  FOR  THE  USE  OF 

MODELS  AND  SIMUUTIONS 


COLONEL  ULITK.PIPLANI,  USA 
LIEUTENANT  COLONEL  JOSEPH  G.  MERCER,  USAF 
i  LIEUTENANT  COLONEL  RICHARD  0.  ROOP,  USAF 
!  SEPTEMBER  1994 


DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 


Systems  Acquisition  Manager’s  Guide 

for  the  use  of 

MODELS  AND  SIMULATIONS 


Report  of  the 
DSMC  1993-1994 
Military  Research  Fellows 

Colonel  Lalit  K.  Piplani,  USA 
Lieutenant  Colonel  Joseph  G.  Mercer,  USAF 
Lieutenant  Colonel  Richard  O.  Roop,  USAF 


September  1994 


PUBLISHED  BY  THE 

DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE  PRESS 
FORT  BELVOIR,  VIRGINIA  22060-5565 


DISCLAIMER 

This  book  was  produced  in  the  Department  of  Defense  (DoD)  school 
environment  in  the  interest  of  academic  freedom  and  the  advance¬ 
ment  of  national  defense-related  concepts.  The  views  expressed  in 
this  book  are  those  of  the  authors  and  do  not  reflect  the  official 
position  or  policy  of  the  DoD  or  those  of  the  United  States  Govern¬ 
ment. 


For  sale  by  the  U.S.  Government  Printing  Office 
Superintendent  of  Documents.  Mail  Slop:  SSOP.  Washington.  DC  20402-9328 
ISBN  0-16-045161-2 


DEDICATION 

In  this,  the  fiftieth  anniversary  year  of  the  Normandy  invasion,  the 
DSMC  Military  Research  Fellows  dedicate  this  guidebook  to  all  who 
have  faithfully  served  the  Armed  Forces  of  the  United  States  of 
America,  and  to  their  families  for  their  support. 


I  Icoeaslcn  For 

!  DTK.  i.;n 
I  Uue.r.r:  -v-.- --3 

i -a - ... 


□  □G3 


I 


ACKNOWLEDGEMENTS 

This  guidebook  is  the  result  of  an  11-month  Military  Research  Fel¬ 
lowship  program  sponsored  by  the  Defense  Systems  Management 
College  (DSMC)  and  Under  Secretary  of  the  Air  Force,  Acquisi¬ 
tion  (SAF/AQ)  and  funded  by  the  Defense  Modeling  and  Simula¬ 
tion  Office  (DMSO). 


I 

I 

I 

I 


IV 


"'The  advantage  we  had  in  DESERT  STORM  had 
three  major  components.  We  had  an  advantage  in 
people,  an  advantage  in  readiness,  and  an  advantage 
in  technology.  ” 


Dr.  William  J.  Perry’s  (as  Deputy  Secretary 
of  Defense)  remarks  to  the  National  Contract 
Management  Association, 

Washington,  DC,  November  18,  1993 


'We  need  to  preserve  that  part  of  the  industrial  base 
which  will  give  us  a  technological  advantage,  but  we 
have  to  do  it  at  a  reduced  cost  and  increased  efficiency 
in  procurement.  ” 


Dr.  William  J.  Perry’s  (as  Deputy  Secretary 
of  Defense)  remarks  to  the  National  Contract 
Management  Association, 

Washington,  DC,  November  18,  1993 


“Simulation  and  modeling  technology  can  be  applied  to 
every  major  DoD  weapon  development  program  to  re¬ 
duce  design  and  production  cost,  improve  performance, 
improve  diagnostics  and  maintenance,  assist  in  better 
and  faster  training  of  personnel,  and  improved  command 
and  control  on  the  battlefield.  ” 


Mrs.  Colleen  Preston’s  (Deputy  Under 
Secretai7  of  Defense  Acquisition  Reform) 
Testimony  to  Congress,  (March  1994) 


NOTICE 


Copies  of  this  report  may  be  obtained 
by  writing  or  faxing  the: 


DEFENSE  SYS  MGMT  COLG 
ATTN  OS  PR 

9820  BELVOIR  RD  STE  G38 
FT  BELVOIR  VA  22060-5565 


Telephone:  (703)  805-2743 
DSN:  655-2743 
Fax:  (703)  805-3857 


TABLE  OF  CONTENTS 


Page 

Preface .  xiii 

Chapter  1  —  Introduction 

1.1  Purpose .  1-2 

1.2  Methodology .  1-2 

1.3  Spectrum  of  Military  Modeling  and  Simulation .  1-3 

1.4  Assumptions .  1-3 

1.5  Acquisition  Environment .  1-4 

1.6  Models  and  Simulations  in  Acquisition .  1-5 

1.7  Objective .  1-6 

Chapter  2  —  Background 

2.1  Today’s  Applications .  2-2 

2.2  Systems  Acquisition  Process .  2-2 

2.3  Requirements  Generation  System .  2-4 

Chapter  3  —  Organization  and  Policy 

3.1  Office  of  the  Secretary  of  Defense  Organization  and  Management 

of  Modeling  and  Simulation .  3-1 

3.2  Joint  Staff  Organization  and  Management 

of  Modeling  and  Simulation .  3-4 

3.3  Department  of  the  Army  Organization  and  Management 

of  Modeling  and  Simulation .  3-6 

3.4  Department  of  the  Navy  Organization  and  Management 

of  Modeling  and  Simulation .  3-10 

3.5  Marine  Corps  Organization  and  Management 

of  Modeling  and  Simulation .  3-13 

3.6  Department  of  the  Air  Force  Organization  and  Management 

of  Modeling  and  Simulation .  3-15 

Chapter  4  —  Classification  of  Models  and  Simulations 

4.1  Constructive  Models  and  Simulations .  4-2 

4.2  Virtual  Simulation .  4-3 

4.3  Live  Simulations .  4-4 

4.4  Hierarchy  of  Models  and  Simulations .  4-5 

4.5  Hybrid  Models  and  Simulations .  4-8 

4.6  Summary .  4-14 


Chapter  5  —  Modeling  and  Simulation  in  Support  of  Acquisition 

5.1  Modeling  and  Simulation  Across  the  Acquisition  Life  Cycle .  5-1 

5.2  Modeling  and  Simulation  in  Support  of  Acquisition  Activities .  5-11 

5.3  Summary .  5-27 

Chapter  6  —  Modeling  and  Simulation  Issues 

6.1  Introduction .  6-1 

6.2  Credibility  of  Models  and  Simulations  (or  “How  can  I 

believe  what  I’m  seeing?”) .  6-1 

6.3  Verification,  Validation  and  Accreditation  (VV&A) .  6-2 

6.4  Integration .  6-6 

6.5  Advanced  Distributed  Simulation  (ADS) .  6-8 

6.6  Standards  for  the  Modeling  and  Simulation  Environments .  6-9 

6.7  Maintaining  Credibility  of  M&S .  6-10 

Chapter  7  —  Management  Considerations  in  Modeling  and  Simulation 

7.1  Introduction .  7-1 

7.2  Survey  of  ACAT  I  &  II  PMOs .  7-1 

7.3  Planning  the  Modeling  and  Simulation  Effort .  7-4 

7.4  Models  and  Simulations  as  Deliverables .  7-11 

Chapter  8  —  Movement  Toward  a  Future  State  of  Modeling  and  Simulation 

8.1  Introduction .  8-1 

8.2  The  Evolution  of  Modeling  and  Simulation  (M«feS)  in  Acquisition .  8-1 

8.3  Virtual  Prototyping  at  the  U.S.  Army  Tank  Automotive  and 

Armament  Research,  Development,  and  Engineerng 

Center  (TARDEC) .  8-2 

8.4  Getting  to  the  Future  State  of  M&S .  8-6 

8.5  Management  Actions .  8-8 

8.6  Some  Application  Best  Practices .  8-8 

8.7  Some  Management  Best  Practices .  8-12 

8.8  Summary .  8-16 

Epilogue 

APPENDICES 

Appendix  A  DoD  Sources  of  Information  for  Modeling  and 

Simulation  in  Weapon  Systems  Acquisition .  A- 1 

Appendix  B  Department  of  the  Army  Sources  of  Information  for 

Modeling  and  Simulation  in  Weapon  Systems  Acquisition .  B-1 


Appendix  C  Department  of  the  Navy  Sources  of  Information  for 

Modeling  and  Simulation  in  Weapon  Systems  Acquisition .  C-1 

f 

Appendix  D  Marine  Corps  Sources  of  Information  for 

Modeling  and  Simulation  in  Weapon  Systems  Acquisition .  D-1 

Appendix  E  Department  of  the  Air  Force  Sources  of  Information  for 

Modeling  and  Simulation  in  Weapon  Systems  Acquisition .  E-1 

Appendix  F  Additional  Sources  of  Information  for  Modeling  and 

Simulation  in  Weapon  Systems  Acquisition .  F-1 

Appendix  G  Modeling  and  Simulation  Templates  for  Use  in 

Weapon  Systems  Acquisition  Activities .  G-1 

Appendix  H  Abbreviations  and  Acronyms  Used  Throughout  this 

Modeling  and  Simulation  Guidebook .  H-1 

Appendix  I  Glossary .  I-l 


LIST  OF  FIGURES 

Figure  1-1.  Systems  Acquisition  Process  Cycle .  1-5 

Figure  2-1.  Three  Major  Decision  Making  Support  Systems .  2-3 

Figure  2-2.  Requirements  Generation  System .  2-4 

Figure  3-1.  Office  of  the  Secretary  of  Defense  Organization  for 

M&S  Management .  3-2 

Figure  3-2.  Joint  Staff  Organization  and  Management  of  Modeling 

and  Simulation .  3-5 

Figure  3-3.  Army  Organization  for  M«feS  Policy  Making  and  Dissemination .  3-8 

Figure  3-4.  Department  of  the  Navy  Modeling  and  Simulation  Management .  3-10 

Figure  3-5.  Marine  Corps  Modeling  and  Simulation  Management .  3-14 

Figure  3-6.  Air  Force  Organization  for  Modeling  and 

Simulation  Management .  3-16 


ix 


A 


Figure  3-7.  Air  Force  Requirements  Generation  System .  3-17 

Figure  4-1.  Classes  of  Models  and  Simulations .  4-2 

Figure  4-2.  Hierarchy  of  Models  and  Simulations .  4-5 

Figure  4-3.  Attributes  and  Uses  of  Models  and  Simulations 

within  the  Hierarchy .  4-9 

Figure  4-4.  Hardware/Software-In-The-Loop  Simulation  (HW/SWIL) .  4-11 

Figure  4-5.  Relationships  of  Models  and  Simulations .  4- 1 4 

Figure  5-1.  Relationship  of  Program  Documents,  Needs  and  Measures .  5-4 

Figure  5-2.  Models  and  Simulations  in  Requirements  Definition .  5-13 

Figure  5-3.  Simulation  Based  Design .  5-17 

Figure  5-4.  Missile  Data  Requirements  and  Test  Assets .  5-20 

Figure  5-5.  Sidewinder  Firing  History .  5-21 

Figure  5-6.  Model-Test-Model  Approach  to  Development .  5-24 

Figure  6-1.  Verification,  Validation  and  Accreditation .  6-2 

Figure  6-2.  M&S  Confidence  Assessment  Scope-of-Evidence .  6-5 

Figure  6-3.  Increasing  Cross-Functional  Integration  of  M&S 

in  Systems  Acquisition .  6-7 

Figure  7-1.  How  Program  Management  Offices  are  Currently  Using 

Modeling  &  Simulation .  7-2 

Figure  7-2.  Summary  of  Modeling  and  Simulation  Needs  Perceived  by  PMOs .  7-3 

Figure  7-3.  Outline  for  a  Simulation  Support  Plan  (SSP) .  7-6 

Figure  8-1.  Virtual  Prototyping  Process .  8-3 

Figure  8-2.  The  Tracked  Vehicle  Workstation:  Expanding  Vehicle 

Simulation’s  Role  In  The  Vehicle  Development  Process .  8-4 

Figure  8-3.  Enabling  Technologies .  8-6 


X 


Figure  8-4.  Notional  Flierarehy  of  Technologies .  8-7 

Figure  8-5.  Combat  System  Engineering  Analysis  Laboratory 

Application  to  System  Development .  8-9 

F’igurc  8-6.  BAT  System  Validation;  Process .  8-14 

Figure  8-7.  BAT  Test  Methodology' .  8-15 

Figure  8-8.  TOMAHAWK  Simulation  Management  Board  Certificate .  8-16 

Figure  B- 1 .  Army  Organizations  Active  in  M&S  for  Acquisition .  B- 1 

F'igure  F-1.  lACs  Associated  with  Military  Missions/Functions .  F-3 

Figure  F-2.  lACs  Associated  with  DoD  Key  Technologies .  F-4 

Figure  G-1.  Modeling  and  Simulation  Application  in  Requirements  Definition ....  G-2 

Figure  G-2.  Modeling  and  Simulation  Application  in  Program  Management .  G-3 

Figure  G-3.  Modeling  and  Simulation  Application  in  Design  and  Engineering .  G-4 

Figure  G-4.  Modeling  and  Simulation  Application  in  Manufacturing .  G-5 

Figure  G-5.  Modeling  and  Simulation  Application  in  Test  and  Evaluation .  G-6 

Figure  G-6.  Modeling  and  Simulation  Application  in  Logistics  Support .  G-7 

Figure  G-7.  Modeling  and  Simulation  Application  in  Training .  G-8 


PREFACE 


I  i  .  V  .  -  I  I  ‘i  •  .  •  ■:ii!’niL'(i  (.•tliii  1^  ol  till  L-'c  iniiil;ii  y  Research  1  el  lows  pai  - 

; i!  ■  ;  iHe  ni  ,i;i  !  i  iia  .ih  a  Sen;,  a  Sei  \  lee  (  ollcire  Research  I'cll', )\cship  program  sjronsoreri 
■  hi.  !  \  ,  S,  e-i  R'elen'v..  lor  Aetiuisit ion  and  Icelinology  ( USD(  AA  f')).  I  he 

'  ,.'l!ov.  xii  , .  00  'L'l  ,!i!  .  101 ,  .  'ses:  I  li  st,  to  pros  ide  protossional  and  military  cduealion 

so  '.ok  0! .  :  I  a  ; u  s  1 1 ,  .ii :  ih.  \i  in\ .  \a\  \  a  till  .Air  l-oree;  and  second,  to  cirnduct  lesearch 
01  .  ■  siioo  ..  i  I  I  .  II  I  ill ,  i  e  si  lo  liie  1  .S,  dc  leiise  acquisition  eommunitv .  In  keeping  with  its 
o'i._  sOo  ,  .  ok  !  o  ;  ''.  ios  man, mement  education  m  tlie  IDoI).  tile  Detense  S\stems 

\!  lii  I  A  o .  oil  0  o  :  I  !  's\!<  ■  0  ,  vipei  ,‘.i  me  w  it!i  the  Harvard  I  nn  ersit\  Cnaduate  School 
ihiMo,s-.  ;a.'  .  r>o.  ms  loi  coiuiuetmg  this  stud\ .  I  he  tellousiiip  program  m- 

^ooi.'O  1  :  ’  .  i.  '.  .:,on(  ro.ei.m'  itn  .M  iii.igement  Decelopmeni  (PMIi))  course  at 

I  I.  ;\0ik  I  :,o  01  o'\ 

(  im  !  .,'s,.:im  h  '  loi ;  ".  as  lui'ded  In  the  1  >e tense  Modeling  and  Simulation  Ot'fice  (  D.MSC  ) ). 
ilk  i‘  o  as ,'[  the  I  osoo.ic  li  n.  s  to  develop  and  produce  a  guidebook  on  modeling  and  simu- 
i  \  Kk  S  I  .is  ;t  ri-!,ik.s  i>‘  svsteiiis  aequisltiou  management.  The  book  will  be  broadb 
vO'p  ioOso;  '•.•i;o\  m.ikeis,  military  dej'iarunetits.  gmernment  ot'fiecs.  research  centers, 
lilk  ,ii  :es  and  n.idcime  mstiuuions. 

1  111,  joarjnist  i  a  tins  u  'e.ireh  gmdel'ook  is  to  priwidc  the  reader  with  current  DoD  initia- 
li'.es,  toiPkomme  [lo.iov  and  guid.mce.  and  irientitieation  of  newly  t'ormed  organizations 
III  NKvS.  1  Ills  loiiik  wili  pi.nide  a  heltci  understanding  of  how  models  are  developed, 
.ip'pliedi  Ml  simul.ii I' ms  .nui  then  '..due  in  analysis;  and  more  importantly,  it  provides  a 
qmeK  releienee  leu  ilie  opplieaiion  of  tliese  tools  in  llie  development  of  weapon  svstems. 
Aisi  .  this  emdel'ook  oilers  lesson--  leained  from  current  acquisition  managers,  advice  on 
m.magcmeiii  pi.iciiees,  jiomlsol  eeuitact  il’OCs)  and  liow  to  access  MifeS  data  bases. 

Iti'img  llic  fast  month  of  the  leseaieh  effort  lime  was  spent  initiating  background  re- 
se.iieh.  de'.euipm;:  a  lese.iieh  plan  .i.id  eonsulling  with  the  DSMC  facultv.The  next  Id- 
weeks  weie  s|ienl  .ittendmg  ihe  ilarvaui  l*MD  course.  Upon  our  return  to  DSMC.  we 
immersed  omselves  m  this  reseaieh  etiort. 

,\s  we  beg. Ill  out  reseaieh.  we  were  cjuiekly  overwlielmed  by  the  amount  of  information  on 
MAS  witli  two  exceptions;  MAS  doeumented  history  and  the  role  of  MAS  in  acquisition. 
Since  our  topic  was  preselected  and  funded  by  DMSO,  we  focused  our  efforts  to  provide  a 
consolidated  guidebook  foi  MAS  in  acquisition  with  little  attention  to  history. 

Id  start  w  ith.  even  a  basic  understanding  of  terms  is  needed.  Wc  will  begin  here  by  provid¬ 
ing  definitions  of  model  and  simulation.  A  model  is  a  physical,  mathematical  or  otherwise 
logical  representation  of  a  svstem,  entitv.  phenomenon  or  process.  The  definition  oisimu- 
hmon  is  twofold:  a  method  for  implementing  ;i  model  over  time:  and  a  technique  for  test- 
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mu.  aiialy/iiig  or  training  in  which  rcal-vvorld  systems  are  used;  or  where  real-world  and 
concc|iUial  systems  are  reproduced  by  a  model. 

The  scope  ot  this  effort  was  another  difficult  aspect.  What  resulted,  with  the  assistance  of 
menihers  from  the  .Acquisition  Task  Forces  on  Modeling  and  Simulation  (ATFM&S),  was 
to  pro\ide  a  quick  refcieiice  type  guide. 

Our  task  w  as  to  produce  this  guidebook  with  a  focus  on  Program  Management.  Difficulties 
arose  in  how  to  approach  the  DoD  s  Systems  Acquisition  Process  while  weaving  M&S  into 
the  process.  The  complexity  was  compounded  because  the  Systems  Acquisition  Process  is 
ciminrised  of  three  separate  but  interfacing  systems:  the  requirements  generation  system; 
the  acquisition  management  system;  and  the  planning,  programming  and  budgeting  sys¬ 
tem  (PPHS). 

The  program  management  world  includes  all  three  sy.stems.  However,  the  only  system  that 
presently  uses  the  entire  realm  of  M&S  is  the  acquisition  management  system.  Therefore, 
this  guidelxHtk  focuses  on  the  acquisition  system  with  some  discussion  of  the  requirements 
generation  system  and  the  PPBS.  For  a  very  good,  quick  tutorial  on  the  DoD  systems 
acquisition  process,  refer  to  the  book  Introduction  to  Defense  Acquisition  Management,  by 
Joseph  H.  Schmoll.' 

Readers  pressed  for  time  may  wish  to  proceed  directly  to  Chapter  Seven  for  Modeling  and 
Simulation  Management  Considerations  and  Chapter  Eight  for  The  Future  State  of  Mod¬ 
eling  and  Simulation  and  Best  Practices.  For  others.  Chapter  One  provides  an  introduc¬ 
tion  to  the  acquisition  environment  and  applicability  of  M&S.  Chapter  Two  provides  gen¬ 
eral  information  about  M&S.  Chapter  Three  provides  a  quick  reference  for  policy  and 
guidance.  Chapter  Four  contains  a  more  detailed  discussion  regarding  The  Classes  of  Mod¬ 
els  and  Simulation.  Chapter  Five  provides  a  view  of  M&S  Applications  from  two  different 
aspects — across  the  acquisition  life  cycle  and  in  support  of  specific  acquisition  activities. 
Chapter  Six  provides  managers  with  information  on  some  of  the  issues  in  the  use  and 
management  of  M&S.  The  Appendices  are  provided  for  expanded  information  about 
each  individual  Service,  as  well  as  additional  M&S  information  sources. 

We  owe  our  gratitude  to  many  people.  During  this  research  effort,  we  have  been  genu¬ 
inely  thankful  for  their  help.  The  faculty  and  staff  at  Harvard  University  and  DSMC  were 
extremely  helpful  with  their  encouragement,  insight  and  support.  A  number  of  people 
have  been  particularly  helpful:  Dr.  Adelia  Ritchie,  Dean  of  the  Research,  Consulting  and 
Information  Division  at  DSMC,  served  as  our  mentor  providing  helpful  advice  and  guid¬ 
ance  throughout  this  research  effort.  Special  thanks  to  Mr.  Dan  Chapla,  CDR  John  Farlin, 
LtCol  Wayd  Weber,  ETC  James  Huskins,  Mr.  Chuck  Cochrane,  CDR  James  Grayson. 
Mr.  Bill  Motley  and  Mr.  Joel  Manary  of  the  DSMC  faculty  for  their  valuable  insights  and 
for  reviewing  our  document.  We  owe  our  gratitude  to  the  DSMC  librarians  for  their  out¬ 
standing  support  throughout  our  effort;  as  well  as  to  the  DSMC  Press  staff  for  their  many 
hours  of  work  to  make  this  a  quality  product. 
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would  be  useful  and  meaningful  to  the  reader. 

There  are  many  others  that  deserve  recognition,  but  in  fairness  to  all,  there  are  too  many  to 
mention.  The  three  research  fellows  would  like  to  thank  all  of  those  interviewed.  We  hope  this 
guidebook  is  as  helpful  to  you  as  you  were  to  us — thank  you. 
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INTRODUCTION 


In  June  1991,  the  Deputy  Secretary  of  Defense 
approved  a  plan  to  strengthen  the  use  of  mod¬ 
eling  and  simulation  (M&S).  He  also  desig¬ 
nated  the  Under  Secretary  of  Defense  for  Ac¬ 
quisition  and  Technology  (USD(A&T)), 
formerly  the  USD(A),  as  responsible  for 
“strengthening  the  use  of  modeling  and 
simulation  in  joint  education  and  training, 
research  and  development,  test  and  evalu¬ 
ation  and  operation  and  cost  analysis.”' 

In  June  1992,  the  Institute  for  Defense 
Analyses  (IDA)  published  a  report  titled  “A 
Review  of  Study  Panel  Recommendations 
for  Defense  Modeling  and  Simulation.”^ 
IDA  reviewed  179  recommendations  made 
by  25  separate  study  panels,  over  a  16-year 
period,  concerning  defense  M&S.  The  De¬ 
fense  Modeling  and  Simulation  Office 
(DMSO),  using  this  document  as  a  concep¬ 
tual  foundation,  reviewed  and  classified  the 
recommendations  and  set  off  to  plan  for  and 
implement  those  that  provided  for  new  and 
extended  applications  for  M&S.  The  DMSO 
especially  focused  on  systems  acquisition  and 
test  and  evaluation  (T&E).The  initiative  in¬ 
stituted  by  the  Deputy  Secretary  of  Defense 
is  now  referred  to  as  the  Defense  Modeling 
and  Simulation  Initiative. 

Some  key  conclusions  drawn  from  the  IDA 


review  were: 

•  Specific  areas,  such  as  the  architectural 
issues  of  interoperability  and  specification 
of  standards,  and  the  life-cycle  support  of 
defense  models  and  simulations,  deserve 
more  attention  and  support. 

•  There  are  many  areas  to  which  defense 
M&S  either  should  be  applied  anew  or  ex¬ 
tended,  especially  those  associated  with  sys¬ 
tems  acquisition. 

•  There  are  substantial  needs  and  oppor¬ 
tunities  for  improving  management  and  co¬ 
ordination  of  defense  M&S  activities. 

In  a  DoD  Inspector  General  (IG)  audit  on 
M&S,  1  Mar  1993,^  the  following  findings 
highlighted  many  shortcomings  in  DoD’s 
ability  to  effectively  and  efficiently  utilize 
models  and  simulations: 

1.  DoD  lacks  adequate  M&S  policy; 

2.  Most  M&S  applications  lack  verifica¬ 
tion,  validation  and  accreditation  (VV&A); 

3.  Significant  effort  is  devoted  to  devel¬ 
oping  “stand  alone”  models  with  no  intent 
to  reuse — costly  duplication; 


4.  Absence  of  a  central  library  resource 
contributes  to  redundant  investment;  and 

5.  FY93  DoD  expenditures  were  esti¬ 
mated  to  be  from  $1.3  to  $1.6  billion — con¬ 
solidation  of  effort  could  have  saved  an  esti¬ 
mated  $800  million. 

The  philosophy  of  M&S  evolves  around 
three  overlapping  areas:  operational  plan¬ 
ning,  acquisition  and  training.  Operational 
planning  helps  utilize  our  equipment  and 
forces  to  best  achieve  our  national  objectives 
and  identify  new  requirements.  Acquisition 
provides  the  items,  systems  and  technology 
the  commander  can  use  to  support  opera¬ 
tional  planning.  Finally,  training  teaches  our 
people  how  to  employ  forces,  use  systems 
and  apply  technology  provided  through  ac¬ 
quisition  to  support  operational  planning. 
The  use  of  M&S  provides  a  comparatively 
inexpensive  way  to  plan,  acquire  and  train. 

Throughout  the  DoD  there  is  an  increasing 
interest  in  addressing  the  problems  identi¬ 
fied  by  the  DoD  IG.  The  formation  of  the 
Acquisition  Task  Force  on  Modeling  and 
Simulation  (ATFM&S),  sponsored  by  the 
DMSO,  represents  a  commitment  of  the 
DoD  to  find  ways  to  optimize  and  fully  uti¬ 
lize  hardware  and  software  tools  and  data 
bases  available  to  all  DoD  agencies,  particu¬ 
larly  program  managers  (PMs). 

1.1  PURPOSE 

The  purpose  of  this  guidebook  is  to  assist 
the  acquisition  community,  by  providing  in¬ 
formation  on  DoD  policy  regarding  M&S, 
identifying  the  existing  M&S  capability  and 
describing  how  M&S  can  be  applied 
throughout  the  acquisition  cycle.  Under¬ 
standing  these  topics  gives  PMs  the  oppor¬ 
tunity  to  pursue  dual-use  technologies,  in¬ 
cluding  commercial-military  integration;  al¬ 


lows  for  faster  and  lower  cost  manufactur¬ 
ing  and  complements  operational  test  and 
evaluation  (OT&E).^ 

1.2  METHODOLOGY 

This  project  was  approached  by  focusing  on 
our  target  customers,  the  program  manag¬ 
ers,  and  what  would  help  make  their  job 
easier. 

This  guidebook  was  developed  using  a  modi¬ 
fied  approach  to  designing  a  technical  docu¬ 
ment.  The  in-process  reviews  (IPRs)  (some¬ 
times  called  murder  boards)  were  conducted, 
first  by  the  fellows,  then  by  the  faculty  at 
DSMC.  Additional  IPRs  were  scheduled 
with  selected  PMs  and  representatives  from 
the  ATFM&S  and  Service  M&S  represen¬ 
tatives,  at  DSMC  prior  to  publishing. 

This  guidebook  was  based  on  the  current 
systems  acquisition  process  and  discusses  the 
capabilities  of  using  M&S  to  enhance  the 
efficiency  of  that  process.  Since  the  use  of 
models  and  simulations  is  functionally  ori¬ 
ented,  this  document  retains  its  applicabil¬ 
ity  to  any  future  modification  of  the  acqui¬ 
sition  process. 

At  the  front  of  this  effort  with  DMSO  fund¬ 
ing,  the  Logistics  Management  Institute 
(LMI) — ^with  support  from  the  Science  Ap¬ 
plications  International  Corporation  (SAIC) 
and  the  MITRE  Corporation  -  sent  surveys 
to  selected  ACAT  I  and  ACAT  II  program 
offices.  These  surveys  were  used  to  deter¬ 
mine  what  models  and  simulations  exist; 
how  they  are  used,  managed  and  analyzed; 
and  what  are  the  “opportunities’’  and  “pit- 
falls”  surrounding  their  use.  The  contrac¬ 
tors  consolidated  the  information  and  pro¬ 
vided  it  to  the  Research  Fellows  and  the 
ATFM&S.  Using  the  surveys  as  a  starting 
point,  data  were  gathered  from  product  cen- 
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ters.  research  centers,  contractors,  using 
commands,  test  centers,  and  training  and 
simulation  conferences  by  site  visits  and  tele¬ 
communications. 

1.3  SPECTRUM  OF  MILITARY 
MODELING  AND  SIMULATION 

The  using  community  of  models  and  simu¬ 
lations  is  broad;  extending  from  operators 
of  weapon  systems  all  the  way  to  analysts  in 
laboratories.  Full  comprehension  of  M&S 
terminology  is  difficult — this  document  will 
not  make  the  reader  an  expert,  but  it  will 
aid  in  the  reader’s  understanding  of  this  very 
complex  topic. 

There  have  been  several  discussions  about 
appropriate  definitions  and  use  of  terms 
throughout  the  DoD;  this  book  is  no  excep¬ 
tion.  However,  after  a  review  of  this  guide¬ 
book  readers  will  have  a  solid  foundation  for 
discussing  and  improving  their  knowledge  of 
M&S. 

There  are  many  ways  to  characterize  M&S. 
The  spectrum  of  M&S  includes  broad  types; 
classes,  hierarchy  and  applications  (func¬ 
tional  areas).  The  three  general  types  of 
models  are:  wargaming;  training;  and  acqui¬ 
sition. 

Wargaming  models  range  from  single  en¬ 
gagement  (one-on-one)  to  joint  theater  level 
campaign  operations.  Training  models  range 
from  single  template  instructional  systems 
to  complex  virtual  reality  simulations.  Ac¬ 
quisition  models  range  from  physical  level 
phenomenon  models  through  engineering 
component  design  tools  to  models  of  sys- 
tems-in-the-end-use-environment. 

Classes  of  models  and  simulations  include 
virtual,  live  and  constructive.  These  are  de¬ 
scribed  in  Chapter  4. 


The  hierarchy  of  models  (usually  depicted  as 
a  pyramid)  represents  the  standard  building 
block  approach  for  interfaces  within  the  mod¬ 
eling  world,  indicating  an  ever  increasing  com¬ 
plexity  from  the  bottom  up.  This  hierarchy  is 
also  discussed  further  in  Chapter  4. 

Applications  of  models  and  simulations  are 
generally  viewed  from  the  functional  per¬ 
spective,  which  are  categorized  as  education, 
training  and  operations;  research  and  devel¬ 
opment;  test  and  evaluation;  analysis;  and 
production  and  logistics.  Applications  of 
models  and  simulations  to  specific  activities 
is  further  discussed  in  Chapter  5. 

This  brief  insight  is  intended  to  whet  the 
readers  appetite  and  show  how  M&S  is  a  vi¬ 
tal  part  of  the  DoD  systems  acquisition  pro¬ 
cess. 

The  advances  being  made  in  computer  hard¬ 
ware  and  software  technologies  are  provid¬ 
ing  increasing  opportunities  to  leverage 
M&S  applications  across  traditional  func¬ 
tional  lines.  For  instance,  some  programs  are 
beginning  to  use  engineering  level  system 
models  in  virtual  combat  environments  nor¬ 
mally  used  by  the  training  community.  Prop¬ 
erly  done,  this  cross-functional  application 
allows  for  the  early  evaluation  of  design 
trade-offs  on  overall  combat  performance. 
Using  existing  combat  simulations  has  two 
payoffs: 

1.  Consistency  of  evaluation  perspective 
between  the  acquisition  and  user  communi¬ 
ties;  and 

2.  Reduction  of  duplication  and  overlap¬ 
ping  simulation  development  efforts. 

1.4  Assumptions 

The  following  assumptions  provide  a  com- 
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mon  starting  point  to  present  M&S  applica¬ 
tions  to  the  systems  acquisition  process. 

•  The  acquisition  community  is  operat¬ 
ing  under  the  acquisition  process  established 
by  DoDD  5000.1,  Defense  Acquisition  and 
DoDI  5000.2,  Defense  Acquisition  Manage¬ 
ment  Policies  and  Procedures,  both  dated 
February  1991. 

•  DoDD  5000.59,  DoD  Modeling  and 
Simulation  (M&S)  Management  definitions 
are  accepted  throughout  the  DoD  as  the 
correct  and  standard  definitions. 

•  The  reader  already  has  a  basic  under¬ 
standing  of  the  systems  acquisition  process 
and  how  its  functional  elements  work. 

•  The  reader  knows  very  little  about  M&S. 

•  The  reader  is  interested  in  knowing  the 
following: 

—  What  people  are  talking  about  when 
referring  to  M&S  being  used  in  acqui¬ 
sition. 

—  How  to  explain  the  use  of  M&S  tools 
in  support  of  a  particular  program. 

—  How  to  make  sure  the  right  models 
and  simulations  are  being  used. 

—  How  to  interpret  what  comes  out  of 
M&S. 

—  How  to  assure  the  use  of  M&S  is 
controlled  with  respect  to  a  particular 
program. 

—  Where  to  go  to  get  help. 

•  This  guidebook  talks  to  acquisition  pro¬ 
grams  under  the  DoD  5000  series  versus  the 


DoD  8000  series,  but  the  concepts  pre.sented 
herein  apply  equally  as  well  to  software  ac¬ 
quisition. 

1.5  Acquisition  Environment 

The  DoD  is  faced  with  a  new  world-wide 
order  of  political,  economic  and  military 
affairs.  National  security  has  many  new 
challenges.  The  Government  is  commit¬ 
ted  to  providing  a  strong  force  capable  of 
effectively  deterring  threats  to  the  United 
States  and  its  allies. 

The  downsizing  of  the  military,  the  reduc¬ 
tion  of  available  resources  and  a  process  that 
takes  more  than  a  decade  to  exploit  advanc¬ 
ing  technologies  and  meet  evolving  require¬ 
ments,  indicate  the  need  for  improved  effi¬ 
ciencies  and  process  improvement. 

1.5.1  Existing  Process 

The  DoD  systems  acquisition  process 
evolved  over  the  years  to  ensure  fair  treat¬ 
ment  to  contractors;  prevent  fraud,  waste 
and  abuse;  keep  a  government  check  on  its 
authority  over  and  demand  on  suppliers;  and 
enhance  socioeconomic  objectives.  The  fi¬ 
nal  acquisition  process  ended  up  being  cum¬ 
bersome;  taking  excessively  long  to  meet 
warfighter  requirements.  In  addition,  the 
administrative  process  drove  DoD  costs 
higher  and  higher. 

1.5.2  Future  Acquisition 

The  challenge  is  to  procure  state-of-the-art 
technology  and  products,  rapidly,  from  reli¬ 
able  suppliers  who  utilize  the  latest  manu¬ 
facturing  and  management  techniques;  as¬ 
sist  United  States  companies  now  predomi¬ 
nantly  dependent  on  DoD  business  to  tran¬ 
sition  to  dual-use  production;  aid  in  the 
transfer  of  military  technology  to  the  com- 
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mercial  sector;  and  preserve  defense-unique 
core  capabilities. 

There  are  many  pros  and  cons  in  the  power 
of  M&S  that  should  be  tempered  with  a 
healthy  dose  of  reality.  All  acquisition  prob¬ 
lems  will  not  be  solved  by  M&S.  However, 
M&S  usage  is  the  key  ingredient  to  success 
in  today’s  environment. 

The  risk  associated  with  the  acquisition  pro¬ 
cess  of  today  can  be  minimized  with  proper 
planning,  use  and  understanding  of  models 
and  simulations.  For  example,  in  a  state¬ 
ment  to  the  U.S.  Senate  Subcommittee  on 


Federal  Services,  Post  Offices  and  Civil 
Service,  March  22,  1994,  the  DUSD(AR) 
stated,  “simulation  offers  the  potential  of 
a  cheaper  and  quicker  way  to  find  failure 
modes  than  does  field  testing.”^ 

1.6  Models  and  Simulations  in  Acquisition 

Models  and  simulations  are  viewed  as  a 
potential  answer  to  many  of  DoD’s  sys¬ 
tems  acquisition  process  problems. 

Models  are  generally  used  to  prove  concepts. 
This  can  be  anything  from  a  mathematical 
calculation  to  building  full-scale  replicas  and 
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submitting  them  to  controlled  environments. 
An  underlying  reason  for  using  models  and 
simulations  is  to  reduce  risk. 

Risk  reduction  is  the  unifying  concept 
throughout  the  entire  systems  acquisition 
process.  It  is  very  logical  to  view  models  and 
simulations  as  tools  to  minimize  risk  to  cost, 
schedule,  performance  and  supportability 
for  the  PM.  From  this  viewpoint  the  value 
added  of  models  and  simulations  can  be  com¬ 
municated. 

An  explanation  of  what  risk  means  in  the  ac¬ 
quisition  system  is  needed  at  this  point. 
When  a  system  is  fielded  (operational),  it  is 
put  there  to  meet  a  particular  requirement 
based  on  a  need.  The  system’s  ability  to  meet 
the  mission  requirements  is  continually 
evaluated.  As  the  system  becomes  outdated, 
the  risk  associated  with  the  system’s  ability 
to  accomplish  the  mission  increases.  A  risk 
assessment  is  conducted  to  determine  if  the 
mission  can  be  accomplished  by  changing  the 
use(s)  of  the  system  (i.e.  tactics);  modifying 
the  system;  or  acquiring  a  new  system.  Once 
the  level  of  risk  is  at  the  point  where  a  major 
modification  or  a  new  system  is  needed,  the 
operational  risk,  through  requirements 
documentation  is  translated  into  program¬ 
matic  risk  and  becomes  shared  with  the  ac¬ 
quisition  community. 

The  acquisition  community  receives  direc¬ 
tion  to  provide  a  system  that  satisfies  require¬ 
ments  evolved  from  the  mission  need.  The 
cost,  schedule,  performance  and  support- 


ability  risks  associated  with  the  acquisition 
process  are  inherent.  The  acquisition  com¬ 
munity  helps  find  the  best  contractor,  man¬ 
ages  the  development  and  reports  progress 
up  through  the  chain  to  aggressively  provide 
the  operational  community  a  system  to  meet 
its  need. 

Since  time  and  resources  are  limited,  this 
usually  creates  a  situation  where  trade-offs 
must  be  made  frequently  using  information 
generated  by  models  aud  simulations  to  get 
the  system  operational  at  an  acceptable  per¬ 
formance  level.  This  completes  the  systems 
acquisition  process  cycle.  Figure  1-1  illus¬ 
trates  this  cycle. 

Why  make  this  trip  around  the  acquisition 
risk  cycle?  We  do  it  to  help  clarify  the  need 
to  use  M&S  to  minimize  risk  through  proper 
use  in  the  systems  acquisition  process. 

1.7  Objective 

The  objective  of  this  guide  is  to  provide  a 
reference  for  acquisition  managers  which  de¬ 
scribes  M&S  policies,  types  of  models  and 
simulations,  applications,  and  key  technical 
and  management  issues.  This  guidebook  is 
intended  for  use  by  program  management 
offices  (PMOs),  acquisition  support  agen¬ 
cies,  policy  makers,  military  departments, 
government  offices,  research  centers,  librar¬ 
ies,  industry  and  academic  institutions.  It 
should  enable  the  manager  to  make  better 
use  of  models  and  to  better  understand  the 
results. 
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BACKGROUND 


The  word  model  brings  to  mind  several  pre¬ 
conceived  mental  descriptions.  Some  re¬ 
member  the  plastic  pieces  of  an  airplane,  car 
or  ship;  all  connected  in  an  orderly  fashion 
to  be  pulled  apart,  assembled,  trimmed  and 
sometimes  painted.  Others  may  remember 
watching  an  old  war  movie  when  the  maneu¬ 
ver  plans  were  drawn  in  the  dirt;  or  as  elabo¬ 
rate  as  a  wood  and  metal  scale  model  of  a 
Nazi  Germany’s  officer  recreation  and  re¬ 
covery  facility  used  for  mission  rehearsal. 
There  are  many  different  descriptions  and 
definitions  of  a  model  throughout  the  DoD. 

Since  World  War  II,  technology  has  ad¬ 
vanced  at  an  ever  increasing  rate.  This  ex¬ 
plosion  of  technology  is  moving  faster  than 
products  can  be  acquired  by  the  acquisition 
community.  With  technology  we  can  ac¬ 
complish  what  was  considered  five  years  ago, 
to  be  impossible.  With  the  utilization  of  mi¬ 
croprocessors,  several  new  spin-off  tech¬ 
nologies  are  popping  up.  These  technolo¬ 
gies  range  from  an  internetted  nervous  sys¬ 
tem  type  communication  link  of  satellites, 
under  sea  and  land  fiber  optic  networks  to 
customized  biological  organisms  used  to  eat 
ocean  oil  spills. 

This  explosion  is  a  dream  come  true  for 
many.  Virtual  reality  (an  interactive,  com¬ 


puter-generated  or  synthetic  environment) 
for  example  is  significantly  changing  our 
lives;  entertainment,  work,  learning,  travel 
and  communications  are  all  incorporating 
virtual  reality.  Information  is  being  moved 
versus  people. 

Benefits  are  also  being  gained  by  utilizing 
virtual  prototypes,  computer  based  simula¬ 
tion  of  systems  with  a  degree  of  functional 
realism.  For  example,  virtual  prototypes  with 
properly  modeled  fluid  dynamics  can  be  used 
in  designing  aircraft,  ships  and  missiles  to 
replace  wind  tunnel  testing:  a  costly  and  time 
consuming  process.’ 

With  this  technological  revolution,  the 
present  acquisition  process  is  impractical. 
This  chapter  will  provide  background  infor¬ 
mation  on  why  the  push  for  M&S;  and  in¬ 
formation  about  some  uses  of  models  and 
simulations. 

In  the  beginning  it  is  absolutely  essential  to 
define  modeling  and  simulation  (M&S)  (de¬ 
fined  previously  in  Preface)  as  it  is  used 
throughout  this  guidebook  (from  DoDD 
5000.59,  DoD  Modeling  and  Simulation  Man¬ 
agement). 

A  model  is  a  physical,  mathematical  or  oth- 


erwise  logical  representation  of  a  system, 
entity,  phenomenon  or  process. 

Simulation  is  twofold;  a  method  for  imple¬ 
menting  a  model  over  time;  and  a  technique 
for  testing,  analyzing,  or  training  in  which 
real-world  systems  are  used,  or  where  real- 
world  and  conceptual  systems  are  repro¬ 
duced  by  a  model. 

2.1  Today’s  Applications 

The  user  community  is  very  broad,  spanning 
not  only  those  involved  in  the  employment 
of  weapon  systems,  but  also  those  involved 
in  all  phases  of  systems  acquisition.  Primary 
developers  of  today’s  models  are  war  col¬ 
leges,  industry,  DoD  laboratories  and  uni¬ 
versities. 

There  is  a  difference  of  opinion  over  mod¬ 
eling  techniques;  the  amount  of  detail  re¬ 
quired;  and  the  value  of  analytical  models, 
simulations,  games  and  field  exercises.  In 
examining  these  differences  of  opinion, 
there  are  a  variety  of  models  and  each  user 
has  a  different  application  in  mind  for  the 
same  model.  This  guide  will  not  cover  all  the 
different  opinions,  nor  provide  an  opinion 
on  the  use  of  one  model  over  another.  The 
decisions  regarding  the  specific  use  of  mod¬ 
els  and  simulations  within  a  given  program 
belong  to  the  readers.  Consideration  should 
be  given  to  the  particular  activities  within  their 
programs,  policies,  and  the  guidelines  and  in¬ 
formation  contained  within  this  guidebook. 

The  user  community  is  divided  into  the  fol¬ 
lowing  functional  areas;  education,  training 
and  operations;  research  and  development; 
test  and  evaluation;  analysis;  and  production 
and  logistics. 

Specific  applications  for  each  of  the  func¬ 
tional  areas  are  broken  out  below. 


•  Education,  training  and  operations  — 
Re-creation  of  historical  battles,  doctrine 
and  tactics  development,  command  and  unit 
training,  operational  planning  and  rehearsal, 
and  wartime  situation  assessment. 

•  Research  and  development  -  Require¬ 
ments  definition,  engineering  design  support 
and  systems  performance  assessment. 

•  Test  and  evaluation  -  Early  operational 
assessment,  development  and  operational 
test  design;  and  operational  excursions  and 
post-test  analysis. 

•  Analysis  -  Campaign  analysis,  force 
structure  assessment,  system  configuration 
determination,  sensitivity  analysis  and  cost 
analysis. 

•  Production  and  logistics  -  System 
producibility  assessment,  industrial  base  ap¬ 
praisal  and  logistics  requirements  determi¬ 
nation. 

This  list  is  not  exhaustive  and  not  mutually 
exclusive,  but  it  certainly  is  representative  of 
the  many  applications  of  M&S  throughout 
the  user  community. 

An  important  aspect  of  the  utilization  of 
models  and  simulations  is  the  application  for 
more  than  one  purpose.  An  example  is  the 
manned  weapon  system  simulation  network 
(SIMNET)  initially  developed  for  training, 
however  now  is  being  used  in  the  develop¬ 
ment  of  doctrine  and  tactics. 

2.2  Systems  Acquisition  Process 

The  goal  of  the  systems  acquisition  process 
is  to  deploy,  in  a  timely  manner,  and  sustain 
an  effective  system  that  satisfies  a  specific 
user’s  need  at  an  affordable  cost.  As  stated 
previously,  an  assumption  was  made  the 
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Three  Major  Decision  Making  Support  Systems 

Systems  Acquisition  Process 


Figure  2-1 .  Three  Major  Decision  Making  Support  Systems 


reader  has  some  understanding  of  the  sys¬ 
tems  acquisition  process  as  it  exists  today. 
However,  the  research  discovered  some  con¬ 
fusion  within  different  portions  of  the  pro¬ 
cess. 

In  order  to  ensure  the  reader  has  a  clear  un¬ 
derstanding  of  the  systems  acquisition  pro¬ 
cess  and  information  on  the  requirements 
generation  system,  this  guidebook  will  briefly 
review  those  areas. 

Readers  that  feel  comfortable  with  the  sys¬ 
tems  acquisition  process  may  wish  to  skip  to 
section  2-3  for  a  review  of  the  requirements 
generation  system — an  often  overlooked, 
but  vital  part  of  the  process. 

The  DoDD  5000.1  establishes  broad  policies 


governing  defense  systems  acquisition  pro¬ 
grams.  It  states  that  the  three  decision-mak¬ 
ing  support  systems  must  interact  and  in¬ 
terface  with  each  other  in  order  for  the  pro¬ 
cess  to  work  effectively.  The  three  systems 
illustrated  in  Figure  2-1  are:  1)  requirements 
generation,  2)  acquisition  management  and 
3)  planning,  programming  and  budgeting 
system  (PPBS). 

The  first  formal  interface  between  the  re¬ 
quirements  generation  system  and  the  ac¬ 
quisition  management  system  occurs  at  mile¬ 
stone  0,  supported  by  the  Joint  Require¬ 
ments  Oversight  Council  (JROC).  Mile¬ 
stone  I  marks  the  first  formal  interface  be¬ 
tween  the  acquisition  management  system 
and  the  PPBS.  This  milestone  also  marks 
program  initiation. 


Requirements  Generation  System 


RECOMMENDATIONS  IMPLEMENTATION 


Figure  2-2.  Requirements  Generation  System 


The  acquisition  management  system,  the  re¬ 
quirements  generation  system  and  the  PPBS 
all  interface  to  meet  decision  points  at  ma¬ 
jor  milestones;  and  during  each  Program  Ob¬ 
jective  Memorandum  (POM)  cycle. 

2.3  Requirements  Generation  System 

Requirements  generation  is  based  on  a  con¬ 
tinuing  process  of  assessing  the  capabilities 
of  the  current  force  structure  to  meet  the 
projected  threat;  while  taking  into  account 
opportunities  for  technological  advance¬ 
ment,  cost  savings  and  changes  in  national 
policy  or  doctrine.  Figure  2-2^  depicts  this 
process. 


The  requirements  generation  system  consists 
of  four  distinct  phases:  definition,  documen¬ 
tation,  validation  and  approval. 

2.3.1  Deflnition 

The  definition  phase  is  an  identification  of  a 
deficiency,  c  mismatch  between  current  ca¬ 
pabilities  and  the  future  (projected) 
threat.This  process  is  known  as  a  mission 
area  assessment  (MAA).  Once  identified, 
these  deficiencies  need  to  be  resolved. 

The  first  alternative  is  to  change  the  organi¬ 
zation;  doc.rine  or  tactics;  or  requirements 
for  additional  training.  These  alternatives 
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arc  called  non-materiel  alternatives.  They 
are  investigated  first  heeause  of  their  rela¬ 
tively  low  cost  and  ease  of  implcmcntulion. 
Should  non-materiel  alternatives  prove  in¬ 
capable  of  resolving  the  deficiency,  the  next 
alternative  is  a  materiel  solution. 

If  a  materiel  solution  is  pursued,  definition 
takes  the  form  of  translating  the  deficiency, 
or  technological  opportunity,  into  a  Mission 
Need  Statement  (MNS).  The  MNS  defines 
the  need  in  broad  operational  capability 
terms,  and  the  format  is  described  in  DoD 
500().2-M,  “ Defense  Acquisition  Management 
Documentation  and  RepoH,  "  February  1991, 
part  2. 

2.3.2  Documentation 

This  documentation  phase  is  the  formal 
preparation  of  documents  that  must  be  co¬ 
ordinated  with  the  affected  authority.  The 
MNS  originator  will  determine  whether  the 
program  i.->  a  potential  major  defense  acqui¬ 
sition  program  (MDAP),  which  requires 
JROC  action. 

Materiel  solutions  are  considered  in  the  fol¬ 
lowing  order  of  precedence; 

1 )  U.se  or  modification  of  an  existing  U.S. 
military  system. 

2)  Use  or  modification  of  an  existing  com¬ 
mercially  developed  or  allied  system  (Non- 
Developmental  Item  (NDI)  approach). 

3)  Cooperative  research  and  development 
program  with  one  or  more  Allied  nations. 

4)  New  Joint-Service  program. 

5)  New  Service-unique  development 
program. 


The  MNS  must  be  coordinated  with  affected 
Serv'ices,  Commanders  in  Chief  (ClNCs). 
and  agcnde.s.  as  well  as  any  necessary  higher 
headquarters,  before  forwarding  to  the  vali¬ 
dation  authority. 

2.3J  Validation 

The  validation  phase  is  a  formal  review  pro¬ 
cess  of  the  documentation  by  an  operational 
authority  (other  than  the  user)  to  confirm 
the  identified  need  and  operational  require¬ 
ment.  As  a  minimum,  the  operational  vali¬ 
dation  authority  reviews  the  MNS,  confirms 
that  a  non-materiel  solution  is  not  feasible, 
assesses  the  Joint  Service  potential,  and  for¬ 
wards  the  MNS  with  a  recommendation  to 
the  milestone  decision  authority  (MDA)  for 
milestone  0  action. 

The  Defense  Intelligence  Agency  (DIA)  will 
validate  the  potential  threat  to  be  countered 
and  certify  intelligence  requirements  for 
potential  ACAT  I  programs. 

For  any  C4  capability,  the  Director,  J -6,  Joint 
Staff,  must  certify  the  need  and  operational 
requirements  for  conformance  to  joint  C4 
policy  and  doctrine,  interoperability,  archi¬ 
tectural  integrity,  and  joint  potential  before 
approval.  Validation  is  a  necessary,  but  not 
sufficient,  step  for  approval. 

2.3.4  Approval 

The  approval  phase  is  the  activity  to  formally 
or  officially  sanction  the  identified  need  and/ 
or  operational  capabilities  described  in  the 
documentation.  Approval  also  certifies  that 
the  requirements  documentation  has  been 
subject  to  the  uniform  process  of  the  DoD 
5000  series  and  the  Chairman  of  the  Joint 
Chiefs  of  Staff  (CJCS)  Memorandum  of 
Policy  No.  77  (MOP  77). 
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Should  the  MNS  be  approved  by  the  JROC, 
it  will  be  forwarded  to  the  Defense  Acquisi¬ 
tion  Board  (DAB)  with  the  recommendation 
that  concept  direction  studies  be  initiated. 
Based  on  a  review  by  the  DAB  Committee 
and  the  DAB,  the  Under  Secretary  of  De¬ 
fense,  Acquisition  and  Technology 
(USD(A&T))  makes  the  final  decision  as  to 
whether  or  not  the  warfighting  deficiency 
warrants  the  initiation  of  concept  direction 
studies.  The  resulting  milestone  0  decision 
is  documented  in  an  Acquisition  Decision 
Memorandum  (ADM),  signed  by  the 
USD(A&T),  the  Defense  Acquisition  Ex¬ 
ecutive  (DAE). 


The  MNSs  for  potential  ACAT  I  level  pro¬ 
grams  which  are  disapproved  are  returned 
to  the  originating  service/agency. 

The  validation  and  approval  authority  for 
ACAT  II,  III,  and  IV  MNSs  are  the  service 
(or  defense  agency)  chiefs  or  CINC  of  the 
respective  Unified  or  Specified  Command. 
Approved  MNSs  for  less  than  ACAT  I  level 
programs  are  forwarded  to  the  component 
acquisition  executive  for  action. 

Each  Service  has  their  own  methodology  for 
meeting  the  guidance  provided  by  the  CJCS 
MOP  77  and  the  DoD  5000  series  documents. 


ENDNOTES 
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ORGANIZATION  AND  POLICY 


3.1  OFFICE  OF  THE  SECRETARY  OF  DEFENSE  ORGANIZATION 
AND  MANAGEMENT  OF  MODELING  AND  SIMULATION 


3.1.1  Background 

Two  offices  under  the  Secretary  of  Defense 
(SecDef)  joined  efforts  to  improve  the 
Department’s  management  and  technology 
in  the  areas  of  modeling  and  simulation 
(M&S).  This  led  to  the  Deputy  Secretary  of 
Defense  (DepSecDef)  approving  a  plan  to 
strengthen  M&S  applications  and  assigning 
the  Under  Secretary  of  Defense  for  Acqui¬ 
sition  and  Technology  (USD(A&T)),  the 
responsibility.This  plan  also  established  the 
Executive  Council  on  Modeling  and  Simu¬ 
lation  (EXCIMS)  and  the  Defense  Model¬ 
ing  and  Simulation  Office  (DMSO). 

The  EXCIMS  is  a  flag  officer  level  advisory 
group  to  USD(A&T)  on  M&S  policy,  initia¬ 
tives,  standards  and  investments.  The  DMSO 
provides  a  full-time  focal  point  for  M&S  ac¬ 
tivities,  and  promulgates  USD(A&T)  directed 
M&S  policy,  initiatives  and  guidance:  promot¬ 
ing  cooperation  among  DoD  components. 


increased  emphasis  on  M&S.  During  his  tes¬ 
timony  before  the  House  Armed  Services 
Committee  in  connection  with  the 
President’s  budget,  March  30, 1993,  the  Sec¬ 
retary  of  Defense  stated  that  DoD  is  plan¬ 
ning  to  undertake  acquisition  reforms  that 
are  even  bolder  than  the  Packard  Commis¬ 
sion  proposed.  Goals  included  streamlin¬ 
ing  and  improving  acquisition,  simplifying 
acquisition  guidance  and  establishing  joint 
civilian-military  requirements.The  use  of 
models  and  simulations  is  viewed  as  a  vital 
aspect  of  acquisition  reform. 

Models  and  simulations  are  powerful  tools 
to  improve  the  acquisition  process — such  as 
improved,  up-front  analysis  and  definition 
of  requirements;  early  simulation  of  the  de¬ 
velopment  process  (design,  test,  manufac¬ 
ture,  support,  etc.);  common  shared  data 
bases;  and  the  potential  for  conducting  mul¬ 
tivariate  analysis  in  the  complex  “what  if” 
world  of  the  program  manager  (PM).  The 
bottom  line  is  M&S  saves  resources. 


Downward  trends  in  the  DoD  acquisition 
budget  have  provided  a  forcing  function  for 


Working  toward  that  end,  the  Director  of  De- 
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Office  of  the  Secretary  of  Defense 
Organization  for  M&S  Management 


Figure  3-1 .  Office  of  the  Secretary  of  Defense  Organization  for  M&S  Management 


fense  Research  and  Engineering  (DDR&E), 
established  the  Acquisition  Task  Force  on 
Modeling  and  Simulation  (ATFM&S),  June 
30, 1993,  for  a  one  year  effort.  The  ATFM&S 
charter  was  to  recommend  actions  that  would 
lead  to  the  more  effective,  integrated  use  of 
M&S  throughout  the  acquisition  process.  Fig¬ 
ure  3-1  illustrates  the  relationships  among 
these  organizations. 

3.1.2  Vision  for  Modeling  and  Simulation 

The  stated  vision  of  the  EXCIMS  reads: 

Defense  modeling  and  simulation  will  provide 
readily  available,  operationally  valid  environ¬ 
ments  for  use  by  DoD  components 

•  to  train  jointly,  develop  doctrine  and  tac¬ 


tics,  formulate  operational  plans  and  assess 
war  fighting  situations 

•  as  well  as  to  support  technology  assess¬ 
ment,  system  upgrade,  prototype  and  full  scale 
development  and  force  structuring. 

Furthermore,  common  use  of  these  environ¬ 
ments  will  promote  a  closer  interaction  between 
operations  and  acquisition  communities  in  car¬ 
rying  out  their  respective  responsibilities.  To 
allow  maximum  utility  and  flexibility,  these 
modeling  and  simulation  environments  will  be 
constructed  from  affordable,  reusable  compo¬ 
nents  interoperating  through  an  open  systems 
architecture.'^ 

More  specifically,  simulated  warfighting  envi¬ 
ronments  can  be  constructed;  allowing  the 
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Services  to  train  the  forces,  plan  operations 
and  assess  the  status  of  actual  operations. 
The  training  would  be  joint,  span  several 
echelons,  involve  large  simulated  forces, 
bridge  large  geographic  regions,  and  involve 
senior  commanders,  as  well  as,  units  and  the 
individual  soldiers.  Status  monitoring  would 
be  based  on  electronic  sand  tables  where 
disposition  of  friendly  and  enemy  forces  can 
be  realistically  portrayed  and  the  conse¬ 
quences  of  those  courses  of  action  simulated. 

Similarly,  these  environments  could  support  the 
acquisition  process.  Simulation  test  beds  would 
allow  new  concepts  to  be  explored  and  system 
requirements  to  be  refined  before  bending  metal 
and  committing  to  expensive  alternate  devel¬ 
opments.  Operational  testing  (OT)  can  be  aug¬ 
mented  by  embedding  live  tests  in  a  broader 
simulated  environment  allowing  a  more  com¬ 
prehensive  systems  test.  Together  these  simu¬ 
lations  can  be  used  to  test  new  systems  or  tech¬ 
nologies;  allowing  the  doctrinal  and  tactical 
implications  of  the  new  capabilities  to  be  ex¬ 
plored  prior  to  any  procurement  or  prototype 
development. 

3.1.3  Office  of  the  Secretary  of  Defense 
(OSD)  Policy  Organizations 

3. 1.3.1  Under  Secretary  of  Defense, 
Acquisition  and  Technology  (USD(A&T)) 

The  USD(  A&T)  is  the  principal  staff  assistant 
and  advisor  to  the  SecDef  for  all  matters  relat¬ 
ing  to  the  DoD  acquisition  systems,  research 
and  development,  production,  logistics,  mili¬ 
tary  construction  and  procurement.  The 
USD(A&T)  issues  plans,  programs,  policies 
and  procedures  for  DoD  M«&:S,  in  coordination 
with  the  DoD  components.  This  office  is  re¬ 
sponsible  for  establishing  DoD-wide  M&S 
goals  and  objectives  and  an  investment  strat¬ 
egy  to  achieve  them. 


3. 1.3.2  Director,  Defense  Research  and 
Engineering  (DDR&E) 

The  DDR&E  is  responsible  to  the 
USD(A&T)  for  matters  pertaining  to  re¬ 
search  and  engineering  planning,  invest¬ 
ment,  implementation  and  development. 
The  DDR&E  is  the  chair  for  the  EXCIMS, 
and  provides  EXCIMS-developed  recom¬ 
mendations  and  advice  to  the  USD(A&T). 

3.1.3.3  Assistant  Secretary  of  Defense,  Pro¬ 
gram  Analysis  and  Evaluation  ASD(PA&E): 

The  ASD(PA&E)  is  the  principal  staff  assis¬ 
tant  to  the  SecDef  for  DoD  PA&E.The 
ASD(PA&E)  is  responsible  for  the  critical  re¬ 
view  of  requirements,  performance  and  life- 
cycle  costs  of  current  and  proposed  weapon 
systems.  Review  of  the  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  falls  directly 
under  the  purview  of  the  ASD(PA&E).  The 
ASD(PA&E)  also  provides  leadership  to  the 
Cost  Analysis  Improvement  Group  (CAIG). 

3. 1.3.4  Director,  Operational  Test  and 
Evaluation  (DOT&E) 

This  office  prescribes  policies  and  procedures 
governing  the  conduct  of  OT&E.  Provides  in¬ 
dependent  assessments  and  reports  as  required 
by  current  statutes.  This  office  also  establishes 
policy  for  the  application  of  M&S  in  support 
of  OT&E. 

3.1.4  Related  Organizations 

3.1.4.1  Executive  Council  on  Modeling  and 
Simulation  (EXCIMS) 

The  EXCIMS  is  the  advisory  forum  for  appli¬ 
cation  and  control  of  M&S,  providing  recom¬ 
mendations  to  the  USD(  A&T)  on  DoD  M&S 
goals,  objectives  and  investment  strategy.  The 
EXCIMS  oversees  development  of  DoD  M&S 
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plans,  programs,  policies  and  procedures. 

3.1.4.2  Defense  Modeling  and  Simulation 
Office  (DMSO) 

The  DMSO  serves  as  the  DoD  focal  point 
for  M&S.  The  DMSO  also  serves  as  the  Ex¬ 
ecutive  Secretariat  for  the  EXCIMS  and  fa¬ 
cilitates  ATFM&S  meetings;  disseminates 
policy  and  guidance  to  the  Services;  and 
maintains  the  Defense  Modeling  and  Simu¬ 
lation  Information  System. 

3.1.4.3  Acquisition  Task  Force  on  Model¬ 
ing  and  Simulation  (ATFM&S) 

A  short  term  (one  year)  acquisition  task 
force,  the  ATFM&S  was  chartered  to  rec¬ 
ommend  action  that  would  lead  to  the  more 
effective,  integrated  use  of  M&S  through¬ 
out  the  acquisition  process. 

3.1.5  OSD  Policy  Documents 


3.1.5.1  DoDD  5000.59,  DoD  Modeling  and 
Simulation  (M&S)  Management  (January  4, 
1994) 

The  DoDD  5000.59  establishes  DoD  policy, 
assigns  responsibilities  and  prescribes  pro¬ 
cedures  for  the  management  of  M&S.  It  es¬ 
tablishes  the  DoD  EXCIMS  and  the  DMSO. 

3.1.5.2  Director,  Operational  Test  and  Evalu¬ 
ation  Policy  for  the  Application  of  Modeling 
and  Simulation  in  Support  of  Operational  Test 
and  Evaluation  (January  24, 1989) 

This  document  provides  policy  guidance  for  the 
use  of  M&S  in  support  of  OT&E.  It  describes 
appropriate  application  of  models  and  simula¬ 
tions,  and  guidelines  to  ensure  credible  results. 

3.1.6  Additional  Information 

Points  of  contact  (POCs)  for  M&S  within  OSD 
and  DoD  agencies  are  found  in  Appendix  A. 


3.2  JOINT  STAFF  ORGANIZATION  AND  MANAGEMENT  OF 
MODELING  AND  SIMULATION 


3.2.1  Background 

Joint  models  and  simulations  (JM&S)  are 
those  models  and  simulations  that  represent 
Joint  and  Service  forces,  capabilities,  mate¬ 
rials  and  services  used  in  the  joint  environ¬ 
ment;  or  by  two  or  more  of  the  military 
Services.The  JM&S  support  assessments,  in¬ 
puts  to  the  Planning,  Programming,  and 
Budgeting  System  (PPBS),  joint  professional 
military  education  and  training,  real-time 
operational  support,  wargaming,  and  recon¬ 
struction  of  operations  and  exercises. 

In  the  past,  DoD’s  wargaming  and  assess¬ 
ment  infrastructure  were  a  heterogeneous 
mixture  of  systems  and  applications.  These 


were  independently  developed,  supported, 
and  operated  by  the  combatant  commands: 
the  Services  and  the  Joint  Staff. 

The  objective  for  the  future  envisions  a  co¬ 
ordinated  development  of  JM&S  to  lever¬ 
age  both  the  considerable  investment  in  ex¬ 
isting  capabilities  and  the  competence  that 
exists  in  widespread  centers  and  users — that 
allow  the  exchange  of  M&S  capabilities 
across  the  JM&S  community.  The  core  con¬ 
cept  is  centralized  management  with  decen¬ 
tralized  execution,  called  Distributed  Models 
and  Simulations  (DMS). 

Tailoring  of  specific  JM&S  capabilities  from 
distributed  M&S  centers  (e.g.  USEUCOM 
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Joint  Staff  Organization  and  Management  of  Modeling 

and  Simulation 


Figure  3-2.  Joint  Staff  Organization  and  Management  of  Modeling  and  Simulation 


Warrior  Preparation  Center,  CFC-Korea 
Battle  Simulation  Center,  Joint  Warfighting 
Center,  USPACOM  Joint  Task  Force  Simula¬ 
tion  Center,  Army  National  Simulation  Cen¬ 
ter,  Air  Force  Blue  Flag,  etc.)  through  fiilly 
operational  distribution  networks  to  support 
specific  requirements  for  a  specific  user  is  a 
concept  called  DMS  pinpoint  support. 

This  netted  pooling  of  expertise  will  (1)  al¬ 
low  M&S  to  evolve  as  the  Service/combat¬ 
ant  command  needs  and  capabilities  evolve, 
(2)  maximize  effectiveness  by  allowing  the 


Service  and  combatant  command  areas  of 
expertise  to  directly  affect  the  tools  available 
for  joint  tasking  and  (3)  maximize  efficiency 
by  enabling  multiple,  non-redundant,  paral¬ 
lel  operations  that  will  help  reduce  the  over¬ 
head  of  traditional  stand-alone  centers.This 
will  also  reduce  competition  for  resources, 
assets  and  applications  assistance. 

3.2.2  Organization 

Figure  3-2^  depicts  how  JM&S  will  receive 
distributed  support  from  the  Services  and 
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participate  in  its  policy  development.  A  con¬ 
figuration  management  (CM)  proponency 
will  be  established  for  each  Service  and  joint- 
community  developed  JM&S.  Centralized 
sourcing  and  maintenance  of  common  data 
bases  required  for  joint  training  and  assess¬ 
ment  will  be  developed  and  distributed  via 
distributed  networks. 

The  Joint  Modeling  and  Simulation  Execu¬ 
tive  Panel  (JMSEP)  chaired  by  the  Deputy 
Director  for  Technical  Operations,  J-8,  pro¬ 
vides  a  forum  for  JM&S  community  inter¬ 
action  and  cooperation. 

3.2  J  Key  Organizations 

The  Vice  Director,  Joint  Staff,  is  the  senior 
information  resource  management  official  and 
supervises  the  JM&S  master  planning  process 
and  approves  the  investment  plan.  All  Staff 
Directorates  play  a  role  in  developing  and  us¬ 
ing  JM&S  within  their  areas  of  responsibility, 
but  the  key  JM&S  organizations  on  the  Joint 
Staff  are  J-7  and  J-8,  both  of  which  provide 
representatives  to  the  DoD  EXCIMS. 

3.2.3.1  J>7,  Operational  Plans  and 
Interoperability  Directorate 

The  J-7  coordinates  development  of  policy  ap¬ 
plications  for  the  use  of  JM&S  in  support  of 
joint  doctrine  development,  joint  training,  and 
joint  exercises.  Through  the  Joint  Warfighting 
Center,  J-7  establishes  liaison  with  combatant 
commands  and  Services  for  JM&S  require¬ 


ments  supporting  the  above  activities. 

3.2.3.2  J-8,  Force  Structure,  Resources  and 
Assessment  Directorate 

The  J-8  coordinates  the  development  of  pwlicy 
applications  for  the  use  of  JM&S  in  all  areas  in¬ 
volving  analysis  and  analytical  methodologies. 
The  J-8  develops  and  maintains  the  JM&S  mas¬ 
ter  plan  and  JM&S  investment  plan.  Chairs  the 
JMSEP  and  formulates  policy  for  the  develop¬ 
ment,  acquisition  and  life-cycle  management  of 
JM&S  in  support  of  joint  applications. 

3.2.4  Key  Documents 

3.2.4.1  Joint  Modeling  and  Simulation 
Summary,  (October  1992) 

This  summary  presents  the  existing  joint  user 
results  of  a  survey  of  the  community  as  a  first 
step  in  the  initiative  to  establish  and  main¬ 
tain  the  JM&S  Master  Planning  Process. 

3.2.4.2  Joint  Modeling  and  Simulation  Evo¬ 
lutionary  Overview,  (February  1994) 

This  document  provides  the  JM&S  commu¬ 
nity  with  a  vision  and  associated  policy  ini¬ 
tiatives  and  technical  activities  to  support  the 
combatant  commands  and  Joint  Staff. 

3.2.5  Additional  Information 

Joint  Staff  POCs  for  M&S  may  be  found  in 
Appendix  A. 


33  DEPARTMENT  OF  THE  ARMY  ORGANIZATION  AND 
MANAGEMENT  OF  MODELING  AND  SIMULATION 


3.3.1  Background 

The  Army  has  used  explicit  representations 
of  combat  systems,  combat  and  other  pro¬ 


cesses  for  a  number  of  years.  Long  before 
the  advent  of  computers,  the  Army  relied 
heavily  upon  information  derived  from  the 
conduct  of  simulations.  History  contains  ex- 


amples  of  planning  and  rehearsing  missions 
using  sand-tables,  and  developing  force 
structure  and  tactics  using  substitutes  for 
weapons  and  weapon  systems  (such  as  jeeps 
for  tanks  and  broom  handles  for  rifles)  dur¬ 
ing  the  Louisiana  Maneuvers  of  1940. 

The  sophistication  of  tools  used  to  model  and 
simulate  combat  systems  and  combat  processes 
evolved  over  the  years.  Many  major  field  and 
command  post  exercises  were  conducted  us¬ 
ing  probability  tables  and  the  rolling  of  die  to 
simulate  the  occurrence  of  events.  Rapid  ad¬ 
vances  in  computer  technology  sped  the  evo¬ 
lution  of  M«feS  into  the  synthetic  environment. 

During  the  evolution  from  predominantly 
physical  representations,  the  Army’s  use  of 
M&S  continued  to  support  a  variety  of  appli¬ 
cations  for  five  major  purposes:  education, 
training,  and  military  operations;  analysis;  re¬ 
search  and  development  (R«&D);  test  and  evalu¬ 
ation  (T&E);  and  production  and  logistics. 

The  Army  introduced  management  of  its  mod¬ 
els  and  simulations  in  the  early  1980s  with 
TRADOC  Regulation  5-4.  Today,  the  Army 
executes  management  of  its  M&S  through  the 
Army  Model  and  Simulation  Management  Of¬ 
fice  (AMSMO)  with  policy  prescribed  in  Army 
Regulation  (AR)  5-11. 

Until  as  recently  as  the  mid-1980s,  the 
Army’s  development  and  use  of  M&S  were 
accomplished  on  an  as-needed  and  as-af¬ 
forded  basis.  In  the  late  1980s,  the  advent 
of  distributed  simulation  technology,  led  by 
the  Advanced  Research  Projects  Agency 
(ARPA),  introduced  the  Army  to  Simulation 
Network  (SIMNET).  The  SIMNET,  coupled 
with  a  downward  trend  in  defense  budgets, 
led  the  Army  to  seek  M&S  applications  si¬ 
multaneously  addressing  more  than  one  of 
the  purposes  mentioned  above. 


In  distributed  simulation  technology,  the 
Army  recognized  the  potential  for  linking 
M&S  of  various  types,  fidelities  and  resolu¬ 
tions,  and  of  establishing  these  linkages  from 
geographically  separated  sites  both  in  CO¬ 
NUS  and  overseas.  In  addition,  the  Army 
has  been  assigned  the  role  of  executive  agent 
for  DoD  in  developing  the  technology  and 
infrastructure  to  support  military  applica¬ 
tions  of  distributed  interactive  simulation 
(DIS). 

The  Army’s  model  and  simulation  hierarchy  is 
defined  at  the  following  levels: 

1.  Theater  and  global  models  and  simula¬ 
tions,  typically  aggregated  at  brigade  level  and 
above. 

2.  Division/corps  level  models  and  simula¬ 
tions,  typically  aggregated  at  maneuver  battal¬ 
ion  level. 

3.  Combined  arms  task  force,  brigade  level 
and  below,  high  resolution  models  and  simu¬ 
lations,  typically  representing  individual 
weapon  systems. 

4.  Individual  item  level  models,  which  in¬ 
clude  those  down  to  a  weapon’s  subsystem  and 
component  level. 

Army  policy  requires  the  use  of  Touchstone 
models  and  simulations  (in  effect,  models  of 
choice)  to  the  maximum  extent  possible.  Com¬ 
manders  are  required  to  ensure  adequate  re¬ 
search  into  the  ability  of  existing  models  and 
simulations,  preferably  Touchstones,  to  meet 
emerging  requirements  prior  to  initiation  of 
an  M&S  development  activity. 

The  Army  has  brought  on  line  and  populated 
th& Models  and  Simulations:  Army  Integrated 
Catalog  (MOSAIC)  as  a  hypertext  tool  avail¬ 
able  to  all  users  and  developers  to  browse 
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through  an  array  of  existing  models  and 
simulations.  The  MOSAIC  offers  a  means  to 
begin  a  search  of  existing  M&S  resources. 
Readers  interested  in  more  detail  onMOSAIC 
are  referred  to  Appendix  B. 

Beginning  in  1994,  the  Army  implemented  a 
policy  that  requires  all  Army  acquisition  strat¬ 
egies  for  ACAT  I  and  II  programs  (subse¬ 
quently  expanded  to  Advanced  Technology 
Demonstrations  (ATD)  and  Top  Level  Dem¬ 
onstrations  (TLD))  to  include  a  Simulation 
Support  Plan  (SSP).  In  this  plan,  the  PM  must 
lay  out  the  functional  requirements  for  M&S 
to  support  engineering  and  combat  develop¬ 
ments,  test  and  evaluation,  training  and  mili¬ 
tary  exercises  to  support  the  program.  The  PM 
must  also  develop  an  M&S  acquisition  strat¬ 
egy  identifying  resources  required  to  bring 
the  M&S  to  fruition. 


3J.2  Organization 

Figure  3-3  depicts  the  relationship  between 
selected  activities  involved  in  making  policy 
for  the  Army’s  management,  development 
and  use  of  models  and  simulations. 

33.3  Key  Organizations 

This  section  describes  the  functions  of  those 
activities  specifically  involved  in  recommend¬ 
ing,  establishing  or  promulgating  M&S 
policy  for  the  Army. 

333. 1  Deputy  U nder  Secretary  of  the  Army 
for  Operations  Research  (DUSA(OR)) 

The  DUSA  (OR)  serves  as  Headquarters,  De¬ 
partment  of  the  Army  (HQDA)  proponent  for 
Army  policy  on  M&S,  and  establishes  proce¬ 
dures  and  policy  to  support  DoD  M&S  efforts. 


Army  Organization  for  M&S 
Poiicy  Making  and  Dissemination 


SEC  ARMY 


AMSEC 


DUSA  (OR) 

ASA  (RDA) 

CSA 

AMSMO 


MACOMS 


[h 


Figure  3-3.  Army  Organization  for  M&S  Policy  Making  and  Dissemination. 
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3.33.2  Army  Model  and  Simulation 
Executive  Council  (AMSEC) 

The  AMSEC  recommends  M&S  policy  guid¬ 
ance  to  the  DUSA(OR);  defines  the  scope 
of,  and  approves  activities  to  be  included  in, 
the  Army  Model  Improvement  Program 
(AMIP)  and  Simulation  Technology  Pro¬ 
gram  (SIMTECH);  and  nominates  projects 
for  incorporation  by  the  DMSO. 

3.3.3.3  Army  Model  and  Simulation 
Management  Office 

The  AMSMO  serves  as  executive  secretariat 
for  the  AMSEC,  and  executive  agent  for  ex¬ 
ecution  of  the  AMIP  and  SIMTECH  programs; 
promulgates  M&S  management  policy  and 
implementing  procedures,  such  as  DA  Pam¬ 
phlet  5-11,  Verification,  Validation  and  Ac¬ 
creditation  (WdiA )  of  Army  Models  and  Simu¬ 
lations;  develops  and  publishes  the  Army 
Model  and  Simulation  Master  Plan;  maintains 
MOSAIC;  and  acts  as  the  Army’s  focal  point 
for  dealing  with  DMSO. 

3.3.4  Army  Policy  Related  Documents 

3.3.4.1  Army  Regulation  (AR)  5-11,  Army 
Model  and  Simulation  Management  Program 
(AMSMP) 

Policy  for  Army  M&S  management  is  pre¬ 
scribed  in  AR  5-11.  The  Army’s  program 
for  management  of  models  and  simulations 
is  formalized  in  AR  5-11.  This  regulation, 
and  the  Army  Model  and  Simulation  Mas¬ 
ter  Plan  take  into  account  emerging  DoD 
initiatives  concerning  management  of  these 
tools,  prescribes  policies  and  responsibilities 
for  their  management  within  the  Army  and 
describes  organizational  responsibilities. 

33.4.2  Department  of  The  Army  Pamphlet 
(DA  PAM)  5-11,  Verification,  Validation,  and 


Accreditation  of  Army  Models  and  Simulations 

The  DA  PAM  5-11  provides  procedures  to 
assist  model  developers,  proponents  and 
sponsors  to  conform  to  the  policies  in  AR  5- 
11.  Specifically,  it  provides  guidance  for  the 
development,  execution  and  reporting  of  all 
W&A  activities.  It  also  addresses  data  cer¬ 
tification  in  reference  to  proper  M&S  use. 

33.43  Army  Model  and  Simulation 
Master  Plan 

This  master  plan  provides  a  blueprint  for  in¬ 
vestment  of  Army  resources  in  an  effective, 
efficient  fashion  in  collaboration  with  other 
members  of  the  DoD  community.  It  also  ad¬ 
dresses  the  environment  which  will  allow 
M&S  technologies  to  advance  the  capabili¬ 
ties  of  a  smaller,  power  projection  Army; 
capable  of  land  force  dominance. 

3.3.4.4  OASA(RDA)  Policy  memorandum, 
‘^Simulation  Support  to  Army  Acquisition,” 
May  24, 1993 

This  memorandum  instructs  the  PM  to  prepare 
a  Simulation  Support  Plan  for  each  ACAT I 
and  II  program  going  for  milestone  review. 

33.4.5  DA  PAM  70-XX,  Army  Acquisition 
Procedures,  Part  5,  Section  H,  “Simulation 
Support  to  Army  Acquisition.”  (Draft) 

Section  H  of  DA  PAM  70-XX  provides  guid¬ 
ance  to  assist  in  the  preparation  and  submis¬ 
sion  of  the  SSP. 

33.5  Additional  Information 

Readers  needing  more  information  on  the 
M&S  functions  performed  by  the  many 
Army  commands  and  agencies  are  referred 
to  Appendix  B  and  the  Army  Model  and 
Simulation  Master  Plan.  Copies  of  that  docu- 
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ment  can  be  obtained  from  the  AMSMO. 


3.4  DEPARTMENT  OF  THE  NAVY  ORGANIZATION  AND 
MANAGEMENT  OF  MODELING  AND  SIMULATION 


3.4.1  Background 

Management  of  M&S  in  the  Navy  originally 
was  focused  on  wargaming  and  tactical  readi¬ 
ness.  In  1992,  management  of  M&S  moved 
to  the  Assessment  and  Affordability  Branch 


(N812)  of  the  Assessments  Division  within 
the  Office  of  the  Deputy  Chief  of  Naval  Op¬ 
erations  (Resources,  Warfare  Requirements 
and  Assessments).  The  Naval  Modeling  and 
Simulation  Program  then  encompassed 
M&S  activities  associated  with  R&D  and 


Department  of  the  Navy  Modeling 
and  Simulation  Management 
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Figure  3-4.  Department  of  the  Navy  Modeling  and  Simulation  Management 
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T&E,  education  and  training,  production 
and  logistics,  and  analysis. 

Space  and  Naval  Warfare  Systems  Com¬ 
mand,  (SPAWAR)  provided  support  includ¬ 
ing  maintaining  a  catalog  for  Naval  Model¬ 
ing  and  Simulation. 

A  Naval  Warfare  Analytical/Modeling  and 
Simulation  Oversight  council  called  “Team 
Mike”  was  formed  to  provide  guidance  and  co¬ 
ordination  for  the  Navy’s  diverse  modeling  and 
simulation  efforts.  This  organization  includes 
representatives  from  the  functional  offices 
within  the  Office  of  the  Chief  of  Naval  Opera¬ 
tions,  as  well  as  throughout  the  Navy  support 
organizations,  i.e.,  Naval  Systems  Commands 
and  Naval  Warfare  Centers. 

3.4.2  Organization 

Because  of  the  wide-ranging  spectrum  of  M&S 
and  breadth  of  the  functional  disciplines  which 
M&S  support;  in  1994  the  Navy  decided  to 
establish  a  management  structure,  shown  in 
Figure  3-4,  which  addresses  oversight;  policy 
and  technical  support;  and  users  of  M&S. 

3.4.3  Key  Organizations 

The  key  organizations  and  activities  included 
in  the  M&S  management  structure  are  de¬ 
scribed  below. 

3.4  J.l  Modeling  and  Simulation  Advisory 
Council 

Membership  in  this  Council  is  from  both  the 
Navy  and  Marine  Corps.  This  Council  will; 

•  Advise  on  the  formulation  of  the  De¬ 
partment  of  the  Navy  (DON)  M&S  vision; 

•  Guide  the  development  of  policy,  co¬ 
ordination  and  technical  support;  and 


•  Promote  the  use  of  DON-wide  com¬ 
mon  support  services. 

3.4J.2  Department  of  the  Navy  Modeling 
and  Simulation  Management  Office 

This  office  consists  of  Navy  and  Marine 
Corps  Policy  and  Coordination  offices  and 
a  Technical  Support  Group. 

3.4.3.3  Navy  and  Marine  Policy 
and  Coordination  Offices 

Each  office  is  responsible  for; 

•  Writing  and  maintaining  their  respective 
Service  Modeling  and  Simulation  Master  Plans 
and  Investment  Strategies; 

•  Coordinating  plans,  programs,  policies 
and  procedures  across  functional  areas;  and 

•  Maintaining  instructions  and  standards 
necessary  to  manage  M&S. 

The  Head  of  the  Navy  Policy  and  Coordina¬ 
tion  Office  resides  within  the  Space  and  Elec¬ 
tronic  Warfare  Directorate  (N6).  The  head  of 
the  Marine  Corps  Policy  and  Coordination  Of¬ 
fice  is  the  Director,  Marine  Corps  Modeling 
and  Simulation  Management  Office 
(MCMSMO). 

3.4.3.4  Technical  Support  Group 

The  Technical  Support  Group  provides  tech¬ 
nical  advice  and  assistance  in  the  execution  of 
M&S  activities  throughout  the  DON.  This 
group’s  responsibilities  include; 

•  Maintaining  an  automated  Navy  M&S 
Master  Catalog  and  acting  as  Naval  POC  for 
input  into  other  DoD  M&S  catalogs; 

•  Providing  management  of  the  DON 


3-11 


VV&A  process; 

•  Advising  in  standards  and  protocols  to 
be  used; 

•  Assisting  in  selection  and  development 
of  M&S  applications; 

•  Building  common  services,  tools  and  data 
bases  for  future  development; 

•  Supporting  development  of  the  multi-ser¬ 
vice  common  simulation  framework  and  nec¬ 
essary  infrastructure  and  tools,  such  as  com¬ 
mon  interface  units,  catalogs,  etc.;  and 

•  Designing  distributed  simulation  exer¬ 
cises. 

The  Technical  Support  Group  is  led  by  the 
Space  and  Naval  Warfare  Systems  Command 
Modeling,  Simulation,  and  Analysis  Group 
(SPAWAR  31)  and  is  augmented  as  necessary 
by  personnel  from  various  Naval  organizations, 
such  as  SYSCOMs,  warfare  centers,  laborato¬ 
ries  and  federally  funded  research  and  devel¬ 
opment  centers. 

3.4.3.5  Executive  Agents 

The  Director,  Space  and  Electronic  Warfare 
(N6)  and  the  Commanding  general.  Marine 
Corps  Combat  Development  Command  act 
as  Executive  Agents;  which  participate  in  the 
Modeling  and  Simulation  Advisory  Council 
and  jointly  provide  oversight  to  the  DON  Mod¬ 
eling  and  Simulation  Management  Office. 

3.4.3.6  Functional  Area  Managers 

Functional  Area  Managers  are  designated 
for  the  following  areas:  acquisition,  research 
and  development;  doctrine  and  training  sys¬ 
tems;  test  and  evaluation;  logistics;  opera¬ 
tions;  training  and  education;  and  assess¬ 


ment.  These  managers  generally  reside 
within  the  Chief  of  Naval  Operatioiis  or  As¬ 
sistant  Secretary  of  the  Navy  organizations 
(the  exception  is  the  Doctrine  and  Training 
Systems  Manager,  who  is  the  Commander. 
Naval  Doctrine  Command).  The  fun<'lional 
area  managers  deal  with  issues  across  a  spec¬ 
trum  of  organizations. 

•  Participate  as  members  of  the  M&S  Ad¬ 
visory  Council. 

•  Provide  vision  for  employment  of  M&S  to 
commands,  facilities  and  organizations  working 
within  their  specified  functional  areas. 

•  Promote  and  support  participation  in 
joint  and  cooperative  research,  develop¬ 
ment,  acquisition  and  operation  of  M&S 
systems:  technologies;  and  capabilities 
within  their  functional  areas. 

•  Participate  in  the  development  of  service 
M&S  master  plans,  investment  plans  and  other 
service  planning  documents. 

3.4.3.7  M&S  Developers  and  Users 

M&S  Developers  and  Users,  such  as  system 
commands,  warfare  centers  and  laborato¬ 
ries,  are  responsible  for  ensuring  compliance 
with  applicable  policies  and  procedures. 

3.4.4  DON  Policy  Related  Documents 

The  Navy  has  prepared  instructions  which 
implement  the  Navy  M&S  policies  and  man¬ 
agement  structure. 

3.4.4.I  SECNAVINST  5200.XX:  Depart¬ 
ment  of  the  Navy  Modeling  and  Simulation 
Program  (Draft) 

This  document  descnbes  the  management 
structure,  organizational  responsibilities  and 
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prescribes  policy  and  guidance  for  the  DON 
M&S  program. 

3AA.2  SECNAVINST  5200.XX;  Verifica¬ 
tion,  Validation  &  Accreditation  of  Models 
and  Simulations  (Draft) 

This  instruction  establishes  policy  and  pro¬ 
cedures,  and  assigns  responsibilities  for 


M&S  VV&A  activities  within  the  DON. 

3.4.5  Additional  Information 

A  list  of  other  DON  organizations,  docu¬ 
ments  and  a  description  of  the  Navy  Mod¬ 
eling  and  Simulation  Catalog  are  contained 
in  Appendix  C. 


3.5  MARINE  CORPS  ORGANIZATION  AND  MANAGEMENT 
OF  MODELING  AND  SIMULATION 


3.5.1  Background 

The  need  to  carry  out  diversified  missions,  and 
train  and  equip  its  forces  within  a  constrained 
DoD  resource  environment  has  led  the  Marine 
Corps  to  examine  more  efficient  methods  to 
define  requirements;  evaluate  solutions;  and 
refine  system  and  equipment  designs.  Stand 
alone  models  and  simulators,  and  advanced 
distributed  simulation  are  recognized  as  pro¬ 
viding  the  basis  for  improving  training,  acqui¬ 
sition  decisions,  test  and  evaluation,  force  struc¬ 
ture  decisions  and  requirements  definition. 

The  Marine  Corps  has  taken  a  series  of  steps  to 
accelerate  employment  of  M&S  technologies. 
These  include:  development  of  a  battle  staff 
training  tool,  the  Marine  Air-Ground  Task 
Force  (MAGTF)  Tactical  Warfare  Simula¬ 
tion  (MTWS);  development  of  an  M&S 
Master  Plan  to  provide  a  coherent  strategy 
for  implementing  the  Marine  Corps  simula¬ 
tion  environment,  and  establishment  of  the 
Marine  Corps  Modeling  and  Simulation 
Office  as  a  central  focal  point  for  M&S. 

3.5.2  Organization 

The  management  of  M&S  in  the  Marine 
Corps  is  centered  in  the  Marine  Corps  Mod¬ 
eling  and  Simulation  Management  Office 


(MCMSMO)  located  within  the  Training  & 
Education  Division  of  the  Marine  Corps 
Combat  Development  Command 
(MCCDC).  To  facilitate  communication,  in¬ 
tegration  and  decision  making,  the  Marine 
Corps  M&S  management  structure  as  shown 
in  Figure  3-5  parallels  the  DoD  M&S  man¬ 
agement  structure  outlined  in  DoD  Direc¬ 
tive  5000.59. 

The  Marine  Corps  M&S  management  structure 
consists  of  an  Executive  Steering  Group,  the 
Marine  Corps  Modeling  and  Simulation  Man¬ 
agement  Office ,  and  the  Marine  Corps  Mod¬ 
eling  and  Simulation  Working  Group 
(MCMSWG). 

3.5.3  Key  Organizations 

The  key  organizations  and  activities  included 
in  the  M&S  management  structure  are  de¬ 
scribed  below. 

3.5.3. 1  Executive  Steering  Group 

A  General  Officer  steering  group  designated 
by  the  Assistant  Commandant  of  the  Marine 
Corps  (ACMC).  The  functions  of  this  group 
include; 

•  Oversight  of  the  Marine  Corps  M&S 
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Marine  Corps  Modeling  and  Simulation  Management 


Figure  3-5.  Marine  Corps  Modeling  and  Simulation  Management 


program; 

•  Approves  Modeling  and  Simulation 
Master  Plan  and  Investment  Strategy; 

•  Approves  and  provides  resources  for 
M&S  initiatives  as  part  of  the  POM;  and 

•  Approves  VV&A  policies  and  proce¬ 
dures. 


3.53.2  Marine  Corps  Modeling  &  Simula¬ 
tion  Management  Office  (MCMSMO) 

The  MCMSMO  is  the  Marine  Corps’  single 
focal  point  for  M«feS;  providing  managerial 
oversight  of  the  Marine  Corps  M&S  pro¬ 
gram,  managing  the  Marine  Corps  M&S 
Master  Plan  and  Investment  Strategy,  and 
supporting  development  of  VV&A  policies 
and  procedures. 

The  Director,  MCMSMO  is  also  the  Director 
of  the  USMC  Policy  and  Coordination  Office 
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supporting  the  DON  Modeling  and  Simula¬ 
tion  Management  Office  as  previously  shown 
in  Figure  3-4. 

3.5.3.3  The  Marine  Corps  Modeling  & 
Simulation  Working  Group  (MCMSWG). 

The  MCMSWG  is  chaired  by  the  head  of 
MCMSMO  and  performs  such  functions  as: 
supports  information  exchange  across  func¬ 
tional  areas;  participates  in  the  development 
of  M&S  policies  and  procedures;  provides 
input  to  the  M«feS  Master  plan  and  Invest¬ 
ment  Strategy;  recommends  M&S  projects 
for  inclusion  in  the  POM;  and  recommends 
VV&A  policies  and  procedures. 

The  MCMSWG  consists  of  representatives 
from  functional  organizations  throughout  the 
Marine  Corps,  and  has  five  standing  commit¬ 
tees:  VV&A,  CM,  functional  integration, 
M&S  Information,  and  data  bases/scenarios. 

3.5.4  Marine  Corps 
Policy  Related  Documents 

The  Marine  Corps  has  prepared  instructions 
which  implement  the  Marine  Corps  M&S  poli¬ 
cies  and  management  structure. 

3.5.4.1  Marine  Corps  Modeling  and  Simu¬ 
lation  Master  Plan,  Marine  Corps  Model¬ 
ing  &  Simulation  Management  Office,  (Draft 


as  of  March  16, 1994) 

This  plan  articulates  the  vision  for  the  Ma¬ 
rine  Corps  simulation  environment;  de¬ 
scribes  the  technical  objectives  for  Marine 
Corps  constructive,  virtual,  and  live  simula¬ 
tions;  and  describes  an  implementation  strat¬ 
egy  including  policy  and  management 
framework. 

Referred  to  within  the  Marine  Corps  Modeling 
and  Simulation  Master  Plan  and  its  appendi¬ 
ces,  but  intended  to  be  published  as  sepa¬ 
rate  documents  are  the  following: 

•  Marine  Corps  Modeling  and  Simulation 
Investment  Plan,  (projected  date  May  1995) 

•  Procedures  and  Guidelines  for  Verifi¬ 
cation,  Validation,  and  Accreditation 

•  Procedures  and  Guidelines  for  Con¬ 
figuration  Management  of  Marine  Corps 
Models  and  Simulators 

•  Marine  Corps  Modeling  and  Simula¬ 
tion  Catalog 

3.5.5  Additional  Information 

Other  Marine  Corps  Organizations  involved 
with  the  development,  management,  and/or 
use  of  M&S  are  listed  in  Appendix  D. 


3.6  DEPARTMENT  OF  THE  AIR  FORCE  ORGANIZATION 
AND  MANAGEMENT  OF  MODELING  AND  SIMULATION 


3.6.1  Background 

The  Air  Force  has  a  long  history  of  M&S 
applications.  From  the  Blue  Box  flight  simu¬ 
lator  called  Pilot  maker  to  the  computer 
aided  design  (CAD)  and  computer  aided 
manufacturing  (CAM)  development  of  the 


B-2:  but  the  road  has  been  bumpy. 

The  Air  Force  Scientific  Advisory  Board 
(SAB)  reported  that  the  Air  Force  has  many 
excellent  examples  of  M&S;  a  growing  need 
for  M&S  to  aid  Air  Force  decision  making; 
demonstrated  low  confidence  in  M&S  by 
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decision  makers;  and  a  lack  of  coherent 
policy  and  structure.^ 

As  the  Services  stand  up  to  adopting  more 
M&S  applications,  significant  organizational 
changes  and  new  initiatives  are  occurring. 

3.6.2  Organization 

The  Air  Force  has  designated  the  Directorate 
of  Modeling,  Simulation,  and  Analysis 
(MS&A)(HQ  USAF/XOM)  as  the  single 
POC  for  M&S  issues  and  activities  within 


the  Air  Force.  They  also  represent  the  Air 
Force  in  joint,  multi-service  and  multi¬ 
agency  M&S  efforts.  The  HQ  USAF/XOM, 
in  conjunction  with  all  Air  Force  user  com¬ 
munities,  is  developing  formal  M&S  policy 
for  the  Assistant  Vice  Chief  of  Staff’s  (HQ 
USAF/CVA)  approval.  Figure  3-6'*  shows  the 
Air  Force’s  organization  for  M&S. 

The  Air  Force  uses  MS&A  at  five  different 
levels: 

1.  Strategic/National  Military  Strategy  level 


Test  Centers 


AEDC 


AFDTC 


AFFTC 


Air  Force  Organization  for  Modeling 
and  Simulation  Management 


HQUSAFtCVA 


HQ  USAF/TE 

HQ  USAF/LG 

- IhQ  USAF/Xci 

SAF/AQ 

ITEPhTERT 

IXOM  1 

AQX  AQT 

JDL 


Logistics  Centers 

OC-ALC 

OOALC 

SA-ALC 

SM-ALC 

WR-ALd 


s 


Product  Centers 
I ASC  I  ESC  I  HSC  I  SMC 


[X^jSPOs] 


ARMYSi 
NAVY  LABS 


FOURMOST 

Lat>oratories 


MODELING  &  SIMULATION  TPIPT 


Figure  3-6.  Air  Force  Organization  for  Modeling  and  Simulation  Management. 
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Air  Force  Requirements 
Generation  System 


Figure  3-7.  Air  Force  Requirements  Generation  System 


2.  Theater/Campaign  level 

3.  Mission  level 

4.  Engagement/Submission  level 

5.  System/subsystem  component  (engi¬ 
neering)  level 

3.6.2.1  Air  Force  Requirements 
Generation  System 

As  stated  earlier,  each  Service  has  their  own 
method  for  meeting  the  Chairman  of  the  Joint 
Chiefs  of  Staff  (CJCS)  Memorandum  of 
Policy  No.  77  (MOP  77)  and  DoD  5000  se¬ 
ries  guidance  for  requirements  generation. 
The  Air  Force  follows  the  same  basic  pro¬ 
cess  as  the  guidance  presents  and  adds  a  few 


management  tools  to  assist  the  internal  co¬ 
ordination  for  all  mission  area  stakeholders. 
Refer  to  Figure  3-7*. 

The  Mission  Area  Assessment  (MAA)  identi¬ 
fies  the  tasks  to  support  mission  objectives  of 
a  strategy-to-task  analysis. 

The  Mission  Needs  Analysis  (MNA)  docu¬ 
ments  deficiencies  in  our  ability  to  accom¬ 
plish  those  tasks.  This  document  never  gets 
to  the  DAB.  It  provides  the  basis  for  the  Op¬ 
erational  Requirements  Document  (ORD). 
This  document  takes  the  task-to-need  view. 

The  Mission  Area  Plan  (MAP)  outlines, 
through  a  series  of  roadmaps,  required  cor¬ 
rective  actions.  The  MAPs  provide  the  need- 
to-capability  plans. 
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3.6J  Key  Organizations 

The  key  organizations  and  activities  included 
in  the  M&S  management  structure  are  de¬ 
scribed  below. 

3.6.3. 1  HQ  USAF/XOM:  Directorate  for 
Modeling  and  Simulation 

This  Directorate  is  the  single  POC  in  the  Air 
Force  for  policy  on  modeling,  simulation  and 
analysis  activity.  Specifically,  provides  sup¬ 
port  to  the  Major  Commands  and  HQ  USAF 
in  modeling,  simulation  and  analysis  that 
involve  Air  Force  plans,  operations  and  op¬ 
erational  requirements. 

3.6J.1.1  XOME:  Evaluation  Support  Divi¬ 
sion  provides  evaluation  support  for  force 
structure  analysis.  Focus  is  on  Studies  and 
Analysis  (S&A)  policy  guidance  on  major 
evaluation  activities,  including  the  COEA 
process  and  the  test  and  evaluation  process. 

3.6.3.1.2  XOMT:  Technical  Support  Divi- 
sion  provides  technology  support;  including 
policy  for  architecture,  standards  and  W&A. 

3.6.3.U  XOMW:  Warfighting  Support  Di¬ 
vision  provides  S&A  policy  guidance  for 
education  and  training  of  Commanders  and 
Battle  Staffs;  real-time  and  interactive. 

3.6.3.1.4  AFSAA:  Air  Force  Studies  and 
Analysis.  The  recent  reorganization  of  HQ 
USAF/XQ  incorporated  Air  Force  S&A 
under  XQM  as  a  Field  Qperating  Agency. 
They  provide  force  application  support  with 
campaign  analysis,  theater  air  defense  infor¬ 
mation,  weapons  and  tactics  information, 
and  force  employment  analysis;  force  en¬ 
hancement  information  and  analysis  for  glo¬ 
bal  deterrence,  global  mobility,  space  and 
C^I,  and  force  support;  and  support  for 
wargaming  exercises. 


3.6J.2  HQAFMC/XRX:  Director  for 
Operational  Requirements 

Provides  M&S  direction/policy  for  AFMC 
acquisition  M&S.  This  office  is  the  single 
AFMC  PQC  for  acquisition  M&S  corre¬ 
spondence  with  the  HQ  USAF  and  SAF  of¬ 
fices  and  AFMC  centers/field  units. 

3.6  J.3  Technical  Planning  Integrated  Prod¬ 
uct  Team  (TPIPT) 

The  ESC/XRP  chairs  this  team  that  is  com¬ 
prised  of  members  from  all  AFMC  product 
centers.  The  M&S  TPIPT  integrates  and  co¬ 
ordinates  AFMC  M&S  activities,  provides 
responses  to  higher  headquarters  and  plans 
for  M&S  tools  that  will  support  acquisition; 
while  being  accessible  and  beneficial  to 
wargaming  and  training. 

3.6.4  Air  Force  Policy  Related  Documents 

3.6.4.1  AFPD  16-10,  Modeling  and  Simula¬ 
tion  (M&S)  Management,  (Draft,  May  1994) 

This  directive  provides  general  policy  for 
M&S  throughout  the  Air  Force  and  assigns 
responsibilities.  It  also  implements  DoDD 
5000.59,  DoD  Modeling  and  Simulation  Man¬ 
agement. 

3.6.4.2  AFI  16-1001,  Verification,  Valida¬ 
tion  and  Accreditation,  (Draft) 

Provides  specific  responsibilities,  proce¬ 
dures,  formats  and  guidelines  for  VV&A. 

3.6.4.3  AFI  16-1002,  M&S  Management, 
(Draft) 

Provides  specific  responsibilities,  proce¬ 
dures,  formats  and  guidelines  on  the  man¬ 
agement  of  M&S. 
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3.6.4.4  AFMCP  800-66,  Air  Force  Materiel 
Command  Models  and  Simulations  Guide, 
(July  1993) 

This  document  provides  guidance  for  M&S 
applications  through  the  life  cycle  of  Air  Force 
weapon  systems.  The  guide  focuses  on  the 
systems  engineering  approach  and  is  de¬ 
signed  to  support  the  Integrated  Weapon 
System  Management  (IWSM)  concept.  The 
scope  is  a  general  overview  of  M&S  appli¬ 
cations,  not  detailed  descriptions. 

3.6.4.5  AFPD  10-6,  Mission  Needs  and 
Operational  Requirements,  (January  1993) 

This  policy  directive  establishes  general 
policy,  assigns  oversight  responsibility, 
implements  DoD  5000  series  documents  into 
the  Air  Force  and  identifies  AFI 10-601  as  a 
companion  to  this  policy  ci  rective. 

3.6.4.6  AFI  10-601,  Mission  Needs  and 
Operational  Requirements  Guidance  and 
Procedures,  (1994) 

The  AF 10-601  provides  guidance  in  prepar¬ 


ing,  validating  and  approving  Mission  Need 
Statements  (MNS),  ORDs  (including  the 
Requirements  Correlation  Matrix  (RCM)), 
and  COEA. 

3.6.4.7  AFPD  10-14,  Air  Force  Policy 
Directive  on  Modernization  Planning, 
(Draft) 

This  policy  directive  establishes  the  modern¬ 
ization  planing  process  to  identify  and  cor¬ 
rect  deficiencies  in  mission  and  functional 
areas. 

3.6.4.8  AFI  10-1401,  Modernization 
Planning  Documentation,  (Draft) 

This  instruction  establishes  responsibilities 
and  defines  major  processes  for  develop¬ 
ment  of  MAPs  used  in  the  modernization 
planning  process. 

3.6.5  Additional  Information 

Additional  information  on  relative  documen¬ 
tation,  key  organizations  and  points  of  con¬ 
tact  are  contained  in  Appendix  E. 
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CLASSIFICATION  OF  MODELS 
AND  SIMULATIONS 


There  were  over  500  Models  and  Simula¬ 
tions  (predominantly  computer  models) 
listed  in  the  Twelfth  Edition  of  the  Joint 
Chiefs  of  Staff  (J-8)  Models  and  Simulations 
Catalog'  alone.  Many  more  models  and 
simulations  are  listed  in  other  catalogs 
throughout  DoD  or  used  in  support  of  spe¬ 
cific  programs  without  being  included  in  any 
catalogs  or  formal  listings. 

The  intent  of  this  guidebook  is  to  describe 
the  forms  that  models  and  simulations  take 
and  their  uses  in  acquisition;  rather  than 
serve  as  an  additional  catalog.^ 

The  discussion  of  the  types  of  models  and 
simulations  will  begin  by  laying  out  a 
framework  of  model  and  simulation  classes. 
This  is  to  show  the  breadth  that  models  and 
simulations  encompass  and  to  provide  the 
acquisition  community  with  an  understand¬ 
ing  of  terminology. 

Before  proceeding  further,  the  reader  is 
cautioned  not  to  become  too  enamored 
with  the  terminology,  nor  should  one  try 
to  fit  every  model  or  simulation  neatly  into 
one  of  the  classes.  In  the  authors’  opin¬ 
ion,  the  lines  among  the  various  classes  of 


models  and  simulations  are  becoming 
blurred-technology  allows  linkage  and 
interoperability  among  the  various  classes  of 
models  and  simulations,  and  human  inter¬ 
actions  can  span  across  all  the  classes.  There¬ 
fore,  one  often  is  not  simply  talking  about  a 
single  model  or  simulation,  but  rather  hy¬ 
brids  formed  from  among  two  or  more 
classes. 

The  Defense  Science  Board  in  1992  defined 
an  appropriate  classification  of  models  and 
simulations  similar  to  that  depicted  in  Fig¬ 
ure  4-1.^  These  classifications  find  useful 
application  to  the  systems  acquisition  pro¬ 
cess  and  have  been  elaborated  upon  in  more 
recent  publications  as  follows: 

“Constructive.  Wargames,  models  and  ana¬ 
lytic  tools ... 

Virtual.  Systems  simulated  both  physically 
and  by  computer.  Real  people  fight  [and 
train]  on  synthetic  battlefields,  interacting 
with  each  other  and  with  artifacts  in  the 
simulation.  Examples  include  individual  air¬ 
craft  [weapon  system]  simulators  and  virtual 
prototypes. 
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Classes  of  Models  and  Simulations 


DESCRIPTION 


CLASSES 


EJi&MekES 


Wargames,  models, 
analytical  tools 


iCONSTRUCTIVE 


•  CEM  (Army) 

•  Brawler  (Air  Force) 

•  ENWGS  (Navy) 

•  MTWS  (Marine  Corps) 

•  Engineering,  Cost,  Support  Models 


Systems  simulated  both  physically 
and  by  computer.  Troops  in 
simulators  fight  on  synthetic 
battlefields. 


*  Aircraft/Vehicle/Ship  simulators 

*  Virtual  Proptotypes  (e.g.  NLOS) 

*  Appended  Trainers 


Operations  with  live  forces 
and  real  equipment  in  the  field 


•  REFORGER 
_ ^  •  Red  Flag 

•  National  Training  Center 

•  Strike  University 

•  Instrumented  prototypes 


Figure  4-1 .  Classes  of  Models  and  Simulations 


Live.  Operations  with  live  forces  and  real 
equipment  in  the  air,  on  the  ground,  on  and 
below  the  sea.  Also  included  are  hardware 
prototypes  on  instrumented  ranges.”** 

This  Chapter  will  concentrate  on  further 
describing  these  three  classes  of  models  and 
simulations;  introducing  the  reader  to  what 
is  termed  hierarchies  of  models  and  simula¬ 
tions,  and  providing  a  discussion  of  hybrid 
applications.  Chapter  5  will  then  describe  the 
application  of  modeling  and  simulation 
(M&S)  across  the  acquisition  life  cycle,  as 
well  as  to  various  acquisition  activities. 

4.1  Constructive  Models  and  Simulations 

The  models  and  simulations  contained 
within  this  class  currently  represent  the  pre¬ 


dominant  form  of  M&S  tools  used  within  or 
in  support  of  a  program  office. 

Constructive  models  and  simulations  consist 
of  computer  models,  wargames  and  analyti¬ 
cal  tools  which  are  used  across  a  range  of 
activities.  At  the  lowest  levels,  they  may  be 
used  for  detailed  engineering  design  and 
costing  or  subsystem  and  system  perfor¬ 
mance  calculations  to  support  development 
of  technical  specifications.  Higher  level  mod¬ 
els  and  simulations  provide  information  on 
the  outcomes  of  battles  or  major  campaigns 
involving  joint  or  combined  forces,  identify 
mission  needs  and  support  operational  ef¬ 
fectiveness  analyse^. 

A  variety  of  constructive  models  may  be  used 
to  represent  a  system  and  its  employment  at 
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different  levels  of  detail,  from  engineering 
physics  of  piece  parts  to  aggregated  combat 
forces  in  a  campaign  analysis. 

Many  constructive  simulations  may  be  per¬ 
formed  either  with  or  without  human  inter¬ 
action.  Without  human  interaction,  they 
might  be  run  in  multiple  iterations  to  pro¬ 
vide  statistical  confidence  in  the  outcomes 
of  the  simulation.  With  human  interaction, 
they  are  often  referred  to  as  wargaming 
simulations  and  are  used  for  battle  staff 
training  or  tactics  development.  The  tactics 
developed  in  such  interactive  simulations 
may  then  be  used  for  establishing  tactics 
within  the  non-interactive  simulations. 

Within  acquisition,  the  uses  of  constructive 
models  and  simulations  include  design  and 
engineering  trade-offs,  cost,  supportability, 
operational  and  technical  requirements  defi¬ 
nition  and  operational  effectiveness  assess¬ 
ments. 

4.2  Virtual  Simulation 

4.2.1  Human-in-the  Loop 

Virtual  simulation  brings  the  system  (or  sub¬ 
system)  and  its  operator  together  in  a  syn¬ 
thetic,  or  simulated  environment.  Although 
this  document  uses  the  term  human-in-the- 
loop  to  represent  these  simulations,  other 
names  include  man-in-the-loop,  warfighter- 
in-the-loop,  or  person-in-the-loop. 

In  a  virtual  simulation,  the  system  may  in¬ 
clude  actual  hardware  which  is  driven  (stimu¬ 
lated)  by  the  outputs  of  computer  simula¬ 
tions.  As  an  example,  a  weapon  system  simu¬ 
lator  may  employ  a  near-real  crew  compart¬ 
ment  with  the  correct  equipment,  controls 
and  display  panels.  A  computer  generated 
synthetic  environment  is  then  displayed  on 
a  screen  in  front  of  the  crew  and  reflected  in 


the  crew  compartment  instrumentation  and 
displays.  Motion  of  the  platform  may  be 
driven  by  the  computer  simulation  to  repre¬ 
sent  the  system  dynamics.  Sounds  of  the  sys¬ 
tem  and  equipment  can  also  be  duplicated. 
The  operators  are  thereby  immersed  in  an 
environment  driven  by  the  simulator  that  to 
them  looks,  feels,  and  behaves  like  the  real 
thing.  During  simulated  missions,  the  crew 
must  operate  the  equipment,  receive  com¬ 
mands  and  control  weapons  just  as  in  a  real 
system. 

Human-in-the-loop  simulations  provide  a 
better  understanding  of  human  reactions 
and  decision  processes  and  man-machine 
interfaces.  They  can  provide  a  platform  for 
crew  training  prior  to  live  exercises  and  tests, 
or  realistic  mission  rehearsal  in  preparation 
for  actual  combat  operations. 

Linked  to  other  simulators,  the  interaction 
of  multiple  weapon  systems  can  be  exam¬ 
ined,  leading  to  changes  in  tactics  or  engage¬ 
ment  rules.  These  simulations  also  provide 
powerful  tools  for  evaluation  of  actual  sys¬ 
tem  hardware  and  software  within  realistic 
environments  for  developmental  programs. 

Human-in-the-loop  simulations  run  in  real 
time,  and  hence  fewer  iterations  may  be  per¬ 
formed  than  with  non-interactive  construc¬ 
tive  simulations. 

4.2.2  Virtual  Prototypes 

A  more  advanced  concept  for  virtual  simula¬ 
tion  is  on  our  doorstep — ^virtual  prototyping. 
In  this  realm,  a  three-dimensional  electronic, 
virtual  mockup,  of  a  system  or  subsystem  al¬ 
lows  an  individual  to  interface  with  a  realis¬ 
tic  computer  simulation  within  a  synthetic 
environment. 

The  representation  is  solely  a  computer 
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simulation  rather  than  actual  hardware  and 
may  be  applied  in  early  prototyping  work  to 
evaluate  concepts;  human-machine-inter- 
faces;  or  to  allow  designers,  logistics  engi¬ 
neers  and  manufacturing  engineers  to  inter¬ 
face  with  the  same  design.  Such  an  approach 
supports  Integrated  Product  and  Process 
Development  (IPPD)  or  concurrent  engi¬ 
neering,  by  providing  a  common  platform 
from  which  all  functional  disciplines  can 
work. 

This  concept  of  the  designer,  operator, 
maintainer  and  manufacturer  ail  interacting 
with  the  same  realistic  three-dimensional 
representation  of  the  system  will  become 
more  prevalent  in  future  acquisition.  For  a 
detailed  description  of  the  capabilities  of  vir¬ 
tual  prototyping,  the  reader  should  examine 
reference  4  at  the  end  of  this  chapter. 

4  J  Live  Simulations 

“Everything  is  simulation  except  combat.”^ 
Live  exercises  where  troops  use  equipment 
under  actual  environmental  conditions  ap¬ 
proaches  real  life  in  combat.  The  live  simu¬ 
lation  provides  a  testing  ground  to  provide 
live  data  on  actual  hardware  and  software 
performance  in  an  operational  environment. 

These  data  also  can  be  used  to  validate  the 
models  and  simulations  used  in  an  acquisi¬ 
tion  program.  This  form  of  simulation  pro¬ 
vides  the  stress  and  decision-making  that  is 
associated  with  human-in-the-loop  simula¬ 
tion.  The  introduction  of  multiple  types  of 
platforms  allows  for  evaluation  of  actual  in¬ 
teraction  and  interoperability.  However,  as¬ 
sembling  the  personnel  and  equipment  and 
conducting  a  live  simulation  is  a  resource 
intensive  enterprise  requiring  time,  funds 
and  people. 

Constructive  and  virtual  simulations  may 


already  have  been  conducted  prior  to  live 
simulations  to  plan  the  tests  or  exercises, 
identify  critical  issues,  rehearse  the  mission 
or  train  the  participants.  They  may  also  be 
used  to  analyze  results  after  the  test,  or  aug¬ 
ment  tests  to  address  scenarios  that  may  not 
be  feasible  due  to  safety  or  environmental 
reasons.  With  the  high  cost  of  live  simula¬ 
tions  (tests),  the  use  of  other,  less  resource 
intensive  forms  of  M&S  is  a  smart  prepara¬ 
tion  tool.  For  example,  an  air-to-air  missile 
in  development  might  be  valued  at  $1M,  and 
a  training  torpedo  firing  could  cost  up  to 
$50K.  As  an  integral  part  of  test  planning 
and  support,  M&S  will  allow  a  program 
manager  (PM)  to  use  such  valuable  assets 
more  efficiently.  For  even  greater  benefits 
to  their  programs,  managers  must  insure  that 
live  simulations  include  adequate  instrumen¬ 
tation.  The  data  thereby  collected  will  serve 
two  important  purposes:  further  validation 
of  models  and  simulations;  and  providing 
“ground  truth”  data  to  support  post-exercise 
debriefs. 

As  the  reader  has  seen  within  the  previous 
discussion,  human  interaction  may  be  a  part 
of  any  of  the  classes  of  M&S.  The  acquisi¬ 
tion  program  manager  may  choose  to  em¬ 
ploy  human  interaction  in  M&S  to  satisfy  two 
functions: 

•  Determination  of  human  decision  mak¬ 
ing  or  logic  patterns  and  their  impact  on  sys¬ 
tem  performance  and  effectiveness.  Simu¬ 
lations  of  any  class  requiring  human  input 
may  serve  this  function. 

•  Identification  and  refinement  of  hu¬ 
man-machine  interfaces.  This  results  from 
simulations  which  allow  for  the  human  to  act 
as  part  of  the  system,  such  as  in  manned 
simulators  or  live  exercises. 

These  three  classes  of  models  and  simula- 


4-4 


tions  (constructive,  virtual  and  live)  may  be 
found  in  varying  levels  of  detail  to  support 
activities  ranging  from  detailed  engineering 
design  to  the  military  utility  of  a  new  system 
or  technology  on  the  battlefield.  To  describe 
the  different  levels  of  models  and  simula¬ 
tions  used  to  support  these  activities,  a  hier¬ 
archy  of  models  and  simulations  is  intro¬ 
duced. 

4.4  Hierarchy  of  Models  and  Simulations 

Models  and  simulations  support  acquisition 
program  activities  ranging  from  design  to 
operational  effectiveness  assessments.  This 


assortment  of  tasks  requires  a  suite  of  mod¬ 
els  and  simulations  with  differing  levels  of 
detail  suited  to  their  particular  application. 
These  models  and  simulations  form  what 
may  be  called  a  hierarchy  of  models  and 
simulations. 

Hierarchies  of  models  and  simulations  are 
described  in  documented  form^  and  also 
found  in  undocumented  form  throughout 
the  DoD. 

The  authors  have  found  that  these  hierar¬ 
chies  are  similar  in  concept  and  vary  only  in 
detail.  Some  extend  to  higher  levels,  includ- 
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ing  national  policy  and  force  structure  plan¬ 
ning,  while  others  extend  down  to  include 
actual  testing.  This  document  describes  a 
hierarchy  that  is  representative  of  those  that 
the  reader  may  come  across  or  use.  This  hi¬ 
erarchy  is  depicted  in  Figure  4-2,  alongside 
a  force  level  and  system  work  breakdown 
structure  (WBS)  to  indicate  the  system  level 
that  corresponds  with  the  level  of  ;inalysis 
to  be  performed. 

The  levels  within  this  hierarchy  include: 

•  Engineering:  for  design,  cost,  manufac¬ 
turing  and  supportability.  Provides  measures 
of  performance  (MOP). 

•  Engagement:  for  evaluating  system  ef¬ 
fectiveness  against  enemy  systems.  Provides 
measures  of  effectiveness  (MOE)  at  the  sys¬ 
tem-on-system  level. 

•  Mission/Battle:  effectiveness  of  a  force 
package,  or  multiple  platforms  performing 
a  specific  mission.  Provides  MOE  at  the 
force-on-force  level. 

•  Theater/Campaign:  outcomes  of  joint/ 
combined  forces  in  a  theater/campaign  level 
conflict.  Provides  measures  of  value  added 
at  the  highest  levels  of  conflict,  sometimes 
called  measures  of  outcome  (MOO). 


4.4.1  Engineering  Level  Models  and  Simu' 
lations 


Engineering  level  models  and  simulations 
are  concerned  with  the  performance; 


producibility;  supportability;  cost  of  compo¬ 
nents,  subsystems  and  systems;  and  the 
trade-offs  associated  therewith.  At  the  engi¬ 
neering  level  there  are  literally  thousands  of 
models  and  simulations  including; 

•  Basic  phenomenology  such  as  aerody¬ 
namics,  fluid  flow,  hydrodynamics,  heat 
transfer,  acoustics,  fatigue,  etc. 

•  Physics  based  models  of  components;  sub¬ 
items;  and  systems  for  design,  performance, 
costing,  manufacturing  and  supportability. 

For  acquisition,  engineering  level  models 
and  simulations  provide  the  basis  for  design 
trade-offs  at  the  component,  subsystem  and 
system  levels;  support  development  of  tech¬ 
nical  design  specifications;  and  support  test 
and  evaluation.  Cost  models  provide  devel¬ 
opment,  production,  and  operations  and 
support  costs.  Support  models  can  include 
reliability,  availability  and  maintainability; 
level  of  repair;  and  provisioning  analyses. 
Manufacturing  models  and  simulations  can 
provide  information  on  producibility  of  a 
particular  design,  as  well  as,  simulation  of 
work  flow  on  the  factory  floor  and  identify 
facilitization  requirements. 

These  engineering  level  models  indicate  per¬ 
formance  capabilities,  often  termed  MOP. 
Examples  of  these  measures  include  radar 
acquisition  range,  miss  distance,  range,  pay- 
load  or  speed.  Such  performance  parameters 
might  be  used  in  the  system  and  develop¬ 
ment  specifications. 

The  representations  of  the  system  in 
higher  level  models  and  simulations 
should  have  their  basis  in  these  engineer¬ 
ing  level  models.  It  is  in  those  higher  level 
models  and  simulations  that  the  actual 
impacts  of  weapon  system  performance  on 
combat  effectiveness  is  evaluated. 
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4.4.2  Engagement  Level  Models  and  Simu¬ 
lations 


Engagement  models  and  simulations  repre¬ 
sent  the  system  in  a  limited  scenario,  such 
as  one-on-one,  few-on-few  or  sometimes 
many-on-many.  This  level  of  simulation 
evaluates  the  effectiveness  of  an  individual 
platform  and  its  weapons  systems  against  a 
specific  target  or  enemy  threat  system.  These 
models  rely  on  system  performance,  kine¬ 
matics  and  sensor  performance  from  the 
engineering  level  models  and  simulations. 
They  provide,  survivability,  vulnerability  and 
lethality  results  for  measures  of  system  ef¬ 
fectiveness  or  for  use  in  higher  level  mod¬ 
els.  Detailed  performance  of  the  subsystems 
such  as  propulsion,  combat  systems,  sensors, 
and  guidance  and  control  may  be  included 
and  evaluated. 

The  outputs  of  engagement  level  models  and 
simulations  indicate  the  effectiveness  of  sys¬ 
tems  and  subsystems  in  an  engagement  sce¬ 
nario  and  are  termed  MOE.  Examples  in¬ 
clude  probability  of  kills,  losses  or  mission 
aborts. 

Acquisition  uses  of  engagement  level  mod¬ 
els  and  simulations  include  identifying  sys¬ 
tem  effectiveness  and  performance  to  sup¬ 
port  requirements  documents  (mission  need 
statement  (MNS)  and  operational  require¬ 
ments  document  (ORD))  and  Cost  and  Op¬ 
erational  Effectiveness  Analyses  (COEA), 
system  level  performance  trade-offs,  test  and 
evaluation  support,  and  evaluation  of  tactics 
changes  and  new  weapon  concepts. 


4.4.3  Mission/Battle  Level  Models  and 
Simulations 


Mission/battle  level  models  and  simulations 
reflect  the  ability  of  a  multi-platform  force 
package  to  accomplish  a  specific  mission 
objective,  such  as  air  superiority,  interdiction 
or  strike  which  might  span  a  period  of  hours. 
It  might  consist  of  an  attacking  force  of 
fighter  and  electronic  warfare  aircraft;  a 
combined  arms  group  attack  or  defense;  or 
carrier  battle  group  operations  consisting  of 
aircraft,  ships  and  combat  systems  against  an 
integrated  air  defense  (e.g.,  Surface-to-Air 
Missiles,  enemy  air  assets). 

In  conjunction  with  human  participation, 
mission/battle  level  simulations  may  be  used 
for  wargaming,  training  and  tactics  develop¬ 
ment. 

The  outputs  of  mission/battle  level  models 
and  simulations  are  MOE.  Typically  at  a 
force  package  level  rather  than  at  the  level 
of  the  individual  platform  and  its  weapon 
system.  Examples  of  these  MOEs  might  in¬ 
clude  loss  exchange  ratios,  probabilities  of 
engagement  or  success  in  achieving  a  spe¬ 
cific  mission  objective. 

The  acquisition  applications  of  such  M&S 
include  analysis  in  support  of  requirements 
for  the  MNS  or  ORD;  operational  effective¬ 
ness  analyses  for  alternatives  evaluation  in 
COEAs;  examination  of  interoperability  and 
eompatibility  issues;  and  in  support  of  test 
and  evaluation. 
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4.4.4  Theater/Campaign  Models  and 
Simulations 


Theater/campaign  models  and  simulations 
represent  combined  force  combat  operations 
and  are  used  to  determine  the  long  term 
outcome  of  a  major  theater  or  campaign 
level  conflict.  Forces  are  often  represented 
as  aggregations  of  lower  level  forces  and  sys¬ 
tems.  These  models  and  simulations  can 
identify  major  deficiencies  in  capabilities  of 
force  structures  and  employment  alterna¬ 
tives. 

Since  these  simulations  usually  encompass 
longer  periods  of  warfare  they  are  more 
likely  to  include  sustainment  representations 
within  the  model.  These  models  usually  re¬ 
quire  the  results  of  lower  level  (engineering, 
engagement  or  mission/battle)  models  and 
simulations  as  inputs  to  generate  the  aggre- 
gated-force  level  capabilities.  Some  may 
even  have  the  capability  to  directly  incorpo¬ 
rate  more  detailed  models  of  specific  systems 
within  their  input  architectures. 

As  with  models  and  simulations  within  other 
levels  of  the  hierarchy,  theater/campaign 
level  simulations  might  be  run  with  human 
interaction.  In  this  interactive  mode,  they 
may  by  used  as  a  wargaming  tool  for  battle 
staff  training  or  tactics  development. 

Whereas  the  engineering  level  models  are 
used  to  determine  actual  performance  val¬ 
ues  for  the  components,  subsystems,  or  sys¬ 
tems  being  modeled;  the  higher  level  mod¬ 
els  in  the  hierarchy  are  used  to  establish 
trends,  identify  driving  factors  and  obtain 


relative  comparisons  of  military  utility 
among  systems  or  groups  of  systems  being 
analyzed. 

The  measures  which  result  from  theater/ 
campaign  level  models  and  simulations  are 
sometimes  termed  outcomes.  Examples  may 
include  force  drawdowns  or  battle  group 
losses,  air  superiority,  and  ground  force 
movements. 

Acquisition  applications  of  theater/campaign 
level  models  and  simulations  include  evalua¬ 
tion  of  force  level  combat  outcomes  in  con¬ 
ducting  Mission  Area  Assessments  (MAA) 
leading  to  development  of  MNS;  support  of 
COEAs;  and  evaluation  of  the  impacts  of  new 
systems  or  operational  concepts. 

The  hierarchy  discussed  above  represents  an 
integrated  framework  for  analysis  of  perfor¬ 
mance,  effectiveness,  tactics  and  doctrine, 
and  conflict  outcomes.  Each  level  in  this  in¬ 
tegrated  framework  is  aimed  at  addressing 
specific  issues  and  relies  on  information  ob¬ 
tained  in  analyses  conducted  at  other  levels. 
Figure  4-3  summarizes  the  primary  attributes 
of  the  various  models  and  simulations  within 
each  level  of  the  hierarchy  along  with  repre¬ 
sentative  examples. 


4.5  Hybrid  Models  and  Simulations 

Up  to  this  point,  the  classes  (constructive, 
virtual  and  live)  and  a  hierarchy  of  models 
and  simulations  have  been  described.  In 
many  applications,  linkage  among  two  or 
more  classes  is  actually  used  resulting  in  hy¬ 
brid  models  and  simulations.  Such  a  hybrid 
might  employ  constructive  analytical  mod¬ 
els  to  represent  a  threat  or  the  kinematics 
of  a  weapon  in  conjunction  with  actual  (live) 
system  hardware  and  software.  Examples  of 
such  hybrid  applications  include  physical 


4-8 


Attributes  and  Uses  of  Models  and  Simulations  within  the  Hierarchy 
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Figure  4-3.  Attributes  and  Uses  of  Models  and  Simulations  within  the  Hierarchy 
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HW/SWIL  Cost  Brawler, ESAMS  Suppressor, EAOSIM  AWSIM,  Thunder 

Factory  simulation 

Support  (LORA,  RAM)  MTWS 


simulations,  stimulators,  hardware/software- 
in-the-loop  (HW/SWIL)  simulations  and 
advanced  distributed  simulations  (ADS). 

4.5.1  Physical  Simulation 

Much  of  the  discussion  within  this  Chapter 
has  focused  on  electronic  representations  of 
systems  and  subsystems.  Physical  simulations 
may  refer  to  a  physical  representation  of  the 
actual  operating  environment  (e.g.  tempera¬ 
ture,  humidity,  shock,  vibration,  etc.)  or 
physical  models  used  in  simulating  the  op¬ 
eration  of  a  system  or  subsystem.  Examples 
of  physical  simulation  include; 

•  Munitions  shock  and  vibration  testing 
using  a  simulated  environment; 

•  Survivability/  vulnerability  evaluations 
using  prototype  structures  and  environmen¬ 
tal  conditions,  such  as  high  speed  airflow 
over  the  structure,  simulating  flight  condi¬ 
tions; 

•  A  firing  impulse  simulator  (a  hydrauli¬ 
cally  operated  ram)  used  to  provide  an  im¬ 
pulse  to  the  gun  barrel  to  physically  repli¬ 
cate  the  shock  of  a  round  being  fired  for 
durability  and  shock  testing  of  the  artillery 
piece;  and 

•  A  test  facility  which  simulates  dynamic 
loads  and  motion  for  evaluation  of  tracked 
and  wheeled  vehicle  suspension  systems. 
Such  a  simulation  might  be  driven  using  live 
simulation  data  obtained  from  vehicles  tra¬ 
versing  over  actual  test  courses. 

These  are  just  a  representative  sample  of 
physical  simulations.  They  can  be  used 
throughout  the  acquisition  process  at  com¬ 
ponent,  subsystem  and  system  level;  for 
evaluating  new  technologies  and  early  pro¬ 
totypes;  replicating  field  failures  and  verify¬ 


ing  fixes.  The  data  collected  from  these  simu¬ 
lations  can  also  be  useful  for  validating  mod¬ 
els  and  simulations  at  all  levels  of  the  hier¬ 
archy. 

4.5.2  Stimulation 

In  many  instances,  the  actual  signals  repre¬ 
senting  the  outside  environment  to  a  test 
article  are  not  available.  These  signals  might 
represent  a  radar  return  from  a  target,  a  sig¬ 
nal  from  another  weapon  system  such  as 
between  a  platform  and  its  weapons,  or  back¬ 
ground  noise  in  the  midst  of  which  the  sys¬ 
tem  must  operate. 

Simulations  are  therefore  used  to  stimulate 
the  test  article  just  as  if  the  outside  signal 
was  present.  These  stimulations  may  come 
from  computer  models,  virtual  simulations 
or  from  live  instrumented  tests.  They  can  be 
“hard-wired”  into  the  system,  or  applied  in 
the  same  manner  as  in  the  real  environ¬ 
ment — e.g.  through  a  sensor  system.  Stimu¬ 
lators  may  be  used  in  acquisition  to  simu¬ 
late  threats  or  other  phenomena  either  in  a 
HW/SWIL  simulation  or  a  live  simulation 
(test)  of  a  weapon  system. 

4.5.3  Hardware/Software-ln-The-Loop 

The  HW/SWIL  simulations  are  often  de¬ 
scribed  as  engineering  level  simulations. 
They  typically  consist  of  multiple  classes  of 
simulations.  The  HW/SWIL  includes  actual 
hardware  and  software,  mathematical  mod¬ 
els,  and  external  stimuli  used  together  to 
demonstrate  the  capability  of  a  system  or 
subsystem  to  operate  within  an  environment 
simulating  actual  conditions.  A  HW/SWIL 
simulation  has  proven  to  be  an  important 
tool  in  system  development,  test  and  opera¬ 
tional  support. 

Figure  4-4’  shows  a  typical  HW/SWIL  simu- 
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Hardware/Software-ln-The-Loop  Simulation 

(HW/SWIL) 


Figure  4-4.  Hardware/Software-ln-The-Loop  Simulation  (HW/SWIL) 


lation  for  a  missile  guidance  system.  In  this 
example,  both  the  guidance  and  control  sec¬ 
tion  are  included  in  the  simulation.  The 
stimulation  to  the  guidance  system  is  a  simu¬ 
lated  target  radar  return  that  the  missile 
would  see  in  operation.  Computer  models 
represent  the  threat  environment  and  pro¬ 
vide  missile  aerodynamics  and  kinematics 
which  determine  tne  target-to-missile  posi¬ 
tioning.  This  HW/SWIL  can  include  the  ac¬ 
tual  hardware,  (sensors  and  processors); 
software  (operational  flight  program);  stimu¬ 
lator  (threat  simulation);  and  mathematical 
models  of  the  missile  dynamics — hence,  a 
linkage  of  multiple  types  of  simulation. 

An  extension  of  the  missile  simulation  noted 
above  is  the  Guided  Weapons  Evaluation 


Facility  (GWEF)  and  the  Preflight  Integra¬ 
tion  of  Munitions  and  Electronic  Systems 
(PRIMES)  facility  at  Eglin  AFB.  The  GWEF 
is  a  HW/SWIL  facility  used  to  evaluate 
weapon  performance  from  launch  to  target 
intercept.  The  PRIMES  is  a  fighter  aircraft 
sized  anechoec  chamber  and  associated  labo¬ 
ratories  supporting  one-on-one  or  many-on- 
one  testing  in  a  flight  simulation  condition. 
Linkage  of  these  two  simulation  facilities 
provides  the  capability  to  simulate  weapons 
on  the  aircraft;  allowing  an  integrated  simu¬ 
lation  of  target  identification,  tracking,  and 
missile  launch  and  flight  through  target  in¬ 
tercept. 

Another  example  of  the  use  of  HW/SWIL 
simulators  is  the  Combat  System  Engineer- 
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ing  and  Analysis  Laboratory  (CSEAL)  at  the 
Naval  Undersea  Warfare  Center.  This  simu¬ 
lation  facility  provides  a  human  interactive 
prototyping  environment  to  support  devel¬ 
opment,  integration,  evaluation  of  combat 
system  technology  products  and  develop¬ 
ment  models  for  submarine  combat  systems. 
It  uses  the  actual  hardware  and  software 
being  evaluated  along  with  realistic  simula¬ 
tions  of  the  ocean  environment,  submarine, 
weapon  systems  and  threat  performance. 
The  CSEAL  allows  for  rapid  prototyping, 
preparation  of  the  prototype  for  live  at-sea 
trials  and  timely  analysis  of  the  at-sea  test 
data. 

The  HW/SWIL  is  important  in  test  and 
evaluation  support,  which  is  discussed  in 
more  detail  in  Chapter  5.  In  development, 
HW/SWIL  simulation  can  be  used  to  dem¬ 
onstrate  new  technology;  evaluate  designs, 
concepts,  and  prototypes;  and  show  the  in¬ 
tegration  of  hardware  and  software.  In  sup¬ 
port  of  test  programs,  the  HW/SWIL  simu¬ 
lations  allow  for  pre-test  simulation  to  iden¬ 
tify  test  conditions.  As  a  risk  reduction  mea¬ 
sure,  they  are  used  for  checkout  of  actual 
hardware  and  software.  These  simulations 
are  also  used  to  conduct  post  test  analysis 
and  to  fill  in  a  test  matrbc  for  conditions 
which  are  either  not  testable  or  for  which  no 
test  assets  are  available.  The  HW/SWIL 
simulations  allow  early  identification  and 
correction  of  developmental  problems  and 
allow  one  to  identify  and  focus  live  tests 
(simulations)  toward  critical  issues. 

In  production  and  operations  support,  the 
HW/SWIL  simulation  can  be  used  for  pro¬ 
duction  lot  sampling,  P^I  studies  and  evalu¬ 
ation  of  changes  in  operational  software  pro¬ 
grams.  With  the  increasing  reliance  on  son- 
ware  within  weapon  systems,  and  regular 
software  changes  to  enhance  or  modify  per¬ 
formance,  HW/SWIL  has  become  a  primary 


tool  in  verifying  the  effects  of  software 
changes  on  performance. 

Typically,  these  simulations  are  run  in  real¬ 
time,  with  sensors,  processors,  guidance  and 
control  systems  operating.  Many  runs  will  be 
performed  to  obtain  statistically  significant 
results  for  each  condition — as  an  example, 
approximately  3200  HW/SWIL  simulation 
runs  were  used  to  examine  a  test  matrix  of 
160  conditions  for  the  Sparrow  missile**  (160 
cases  x  20  simulation  runs  per  case  =  3200 
simulation  runs).  Over  a  period  of  sbc  years 
(1976-1982),  a  total  of  39,300  simulation  runs 
were  conducted  to  support  ongoing  missile 
development  and  conduct  parametric  evalu¬ 
ations. 

One  challenge  for  the  program  office  or 
technical  support  activity  comes  about  when 
longer  range  systems  are  evaluated.  Since 
these  simulations  run  in  real  time,  longer 
range  systems  require  more  run  time:  po¬ 
tentially  limiting  the  number  of  simulation 
runs  that  can  be  performed.  Methods  to 
evaluate  only  critical  segments  of  a  mission 
might  need  to  be  pursued  requiring  in-depth 
examination  of  initial  conditions  for  each 
segment. 

Other  management  considerations  which 
must  be  integrated  into  the  planning  for  use 
of  HW/SWIL  simulations  include: 

•  Requiring  appropriate  interfaces  (e.g. 
for  signal  inputs  and  outputs)  designed  into 
the  weapon  system  component  so  that  it  is 
compatible  with  the  simulation  facility  and 
the  desired  data  can  be  accessed; 

•  Facility  and/or  weapon  system  provi¬ 
sions  to  allow  repeated  operation  in  simula¬ 
tions  resulting  in  usage  far  exceeding  its 
planned  mission  time.  This  may,  for  example, 
include  provisions  for  simulation  facility  sup- 
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plied  cooling  of  weapon  electronics;  and 

•  Simulation  facility  development  efforts 
which  may  be  necessary  to  praperly  support 
new  weapon  system  technologies  or  new 
threats. 

4.5.4  Advanced  Distributed  Simulation 
(ADS) 

The  ADS  is  an  emerging  form  of  simulation 
that  has  demonstrated  the  ability  to  link  dif¬ 
ferent  types  of  simulators  at  dispersed  loca¬ 
tions;  permitting  the  simulators  and  their 
crews  to  conduct  operations  on  the  same 
simulated  battlefield  environment. 

The  term  distributed  refers  to  geographically 
separated  simulations,  each  hosted  on  its 
own  computer  without  a  central  computer. 
These  simulations  are  interactive,  indicating 
that  simulations  or  simulators  are  linked  so 
that  they  can  act  upon  one  another  in  a  com¬ 
mon  environment  (e.g.  terrain,  ocean, 
weather,  etc.).  The  linked  simulations  may 
be  any  combination  of  constructive,  virtual 
and  live;  and  likely  to  include  human-in-the- 
loop  simulations.  The  infrastructure  within 
which  such  distributed  simulation  takes 
place  is  termed  Distributed  Interactive 
Simulation  (DIS),  which  is  discussed  further 
in  Chapter  6. 

Acquisition-related  uses  of  distributed  simu¬ 
lation  include  advanced  concept  and  tech¬ 
nology  evaluations  within  a  simulated  battle¬ 
field  environment  leading  to  requirements 
definition.  Performance  and  requirements 
trade-offs  may  also  be  conducted  to  define 
system  performance  objectives,  thresholds, 
manpower  constraints,  critical  system  char¬ 
acteristics  and  man-machine  interfaces.  Ad¬ 
vanced  technology  demonstrators,  used  in 
conjunction  with  simulations  and  live  exer¬ 
cises,  may  allow  evaluation  of  technology, 


manufacturing  and  interoperability  with 
other  systems  prior  to  major  commitment  of 
resources.  Distributed  simulation  might  also 
support  test  and  evaluation  planning,  test 
operator  training,  scenario  development,  ex¬ 
ecution  (e.g.  with  additional  simulated 
forces)  and  post-test  evaluation  of  results.'* 

To  date,  use  of  distributed  simulation  in  ac¬ 
quisition  has  been  limited.  Two  examples 
cited  in  1993  by  the  Defense  Science  Board 
include  an  operational  test  of  the  Non  Line- 
Of-Sight  (NLOS)  missile  and  prototyping  of 
M-1  main  battle  tank  upgrades.'" 

The  NLOS  is  a  vehicle  mounted  missile 
which  underwent  early  operational  testing 
to  assess  the  concept,  requirements  and 
hardware;  along  with  its  adaptation  by  forces 
in  engagements  against  helicopters.  This 
testing  required  thirteen  months  and  $15.5 
million.  A  parallel  evaluation  with  the  use 
of  distributed  simulation  using  an  NLOS 
simulator  and  helicopter  simulators  at  two 
different  locations  and  operating  on  a  com¬ 
mon  terrain  database  was  subsequently  con¬ 
ducted.  This  evaluation  was  conducted  in 
only  3  months  at  a  cost  of  $2  million. 

In  1984,  a  test  bed  (a  live  simulation)  of  an 
upgraded  M-1  tank  was  undertaken.  After  two 
years’  time  and  expenditure  of  $40  million,  the 
simulation  was  not  yet  functional.  At  that  time, 
the  simulation  was  shifted  to  a  modified  air¬ 
craft  dome  and  successfully  completed  in  six 
months  at  a  cost  of  $1  million. 

In  the  future,  distributed  simulation  will  cer¬ 
tainly  increase  in  capability  to  support  the 
acquisition  process.  There  has  been  an  in¬ 
creasing  emphasis  in  the  M&S  community 
on  linkages,  within  and  among,  classes  of 
models  and  simulations.  There  is  a  trend 
toward  local  networks  within  given  research 
and  development  centers  along  with  the  aim 
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of  making  simulations  and  such  networks  com¬ 
patible  with  the  DIS  environment  to  facilitate 
distributed  simulation.  To  take  advantage  of 
this  future  ADS  environment,  managers  in 
their  M&S  planning  should  consider: 

•  The  potential  use  of  the  ADS  capabil¬ 
ity  in  support  of  their  program  activities; 

•  The  likely  requirement  to  make  repre¬ 
sentations  of  their  system  available  for  use 
in  others’  scenarios;  and 

•  The  incorporation  of  appropriate  data 
communication  standards  to  address  inter¬ 


operability  requirements  of  models  and  simu¬ 
lations  they  develop. 

4.6  Summary 

This  chapter  provided  an  overview  of  the 
classes  of  models  and  simulations  which  may 
be  used  during  the  acquisition  life  cycle.  As 
shown  in  Figure  4-5,  a  program  will  likely 
employ  a  suite  of  models  and  simulations. 
The  engineering  level  models  will  provide 
measures  of  performance  along  with  design, 
cost,  producibility  and  supportability  infor¬ 
mation  for  components,  subsystems  or  sys¬ 
tem.  The  military  utility  of  the  system  is 
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evaluated  within  engagement  and  mission/ 
battle  level  models  which  indicate  MOE.  At 
the  highest  level,  the  outcomes  of  major  con¬ 
flicts  involving  combined  forces  are  evalu¬ 
ated  within  theater/campaign  level  models. 
Human-in-the-loop,  virtual  simulators  and 
virtual  prototypes  may  provide  information 
at  all  levels  of  the  hierarchy.  As  in  any  analy¬ 
sis,  the  input  data  and  assumptions  are  ma¬ 
jor  drivers  in  the  results  of  all  simulations. 


Just  some  of  the  system,  environment,  threat 
and  tactics  data  requirements  are  shown  in 
Figure  4-5.  The  PM  should  remember  that 
there  is  no  single  model  or  simulation  that 
will  suit  all  of  a  program’s  needs.  Each  model 
or  simulation  has  a  specific  purpose  for 
which  it  is  intended  and  will  provide  infor¬ 
mation  at  the  requisite  level  of  detail  to  sup¬ 
port  specific  activities  during  the  program 
life  cycle. 


ENDNOTES 


1.  Force  Structure,  Resource,  and  Assessment  Di¬ 
rectorate  (J-S),  The  Joint  Staff.  (1992).  Catalog 
of  Wargaming  and  Simulation  Models  (12th  ed.). 
Washington  DC. 

2.  Note:  For  those  readers  who  need  further  infor¬ 
mation  on  specific  models  and  simulations,  the 
appendices  of  this  guidebook  contain  points  of 
contact  within  each  Service,  and  sources  of  mod¬ 
els  and  simulations  catalogs. 

3.  Office  of  the  Under  Secretar>'  of  Defense  for 
Acquisition.  ( 1993).  Impact  of  Advanced  Distrib¬ 
uted  Simulation  on  Readiness,  Training  and 
Prototyping.  Washington,  DC;  Report  of  the  De¬ 
fense  Science  Board  Task  Force  on  Simulation, 
Readiness,  and  Prototyping. 

4.  Garcia,  A.,  Gocke,  R.  Jr.,  Johnson,  N.  (1994)  Vir¬ 
tual  Prototyping,  Report  of  the  DSMC  1992-1993 
Military  Research  Fellows,  Ft.  Belvoir,  VA:  DSMC 
p34. 

5.  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition.  (1993).  Impact  of  Advanced  Distrib¬ 
uted  Simulation  on  Readiness,  Training  and 
Prototyping.  Washington,  DC;  Report  of  the  De¬ 
fense  Science  Board  Task  Force  on  Simulation, 
Readiness,  and  Prototyping. 


6.  AFMCP  800-66:  AFMC  Models  and  Simulations 
(M&S)  Guide  (July  1993).  Wright-Patterson  AFB, 
OH:  HQ  Air  Force  Materiel  Command;  and  Mili¬ 
tary  Operations  Research  Society.  ( 1989).  Militaiy 
Modeling  (2d  ed.)  W.  P.  Hughes,  Jr.  (Ed). 

7.  Adapted  from  notes  provided  by  Emil  J.  Eichblatt, 
Naval  Air  Warfare  (Weapons  Division).  Point 
Mugu.  CA. 

8.  Eichb\attE.J.Ji.( 1983).  Performance  Simulation 
in  support  of  Test  and  Evaluation.  Point  Mugu,  CA: 
Pacific  Missile  Test  Center  (now  is  Naval  Air 
Warfare  Division,  Weapons  Division). 

9.  Distributed  Interactive  Simulation  Master  Plan. 
(Draft,  1994).  Washington.  DC:  HQDA,  Office 
of  the  Assistant  Deputy  Chief  of  Staff  for  Opera¬ 
tions  and  Plans. 

10.  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition.  (1993).  Impact  of  Advanced  Distrib¬ 
uted  Simulation  on  Readiness,  Training  and 
Prototyping.  Washington,  DC:  Report  of  the  De¬ 
fense  Science  Board  Task  Force  on  Simulation, 
Readiness,  and  Prototyping. 


4-15 


pm 

5 

MODELING  AND  SIMULATION  IN 
SUPPORT  OF  ACQUISITION 


The  use  of  modeling  and  simulation  (M&S) 
within  acquisition  is  a  multi-dimensional 
activity  which: 

•  supports  the  milestone  decision  pro¬ 
cess; 

•  supports  multiple  communities  (opera¬ 
tor,  developer,  designer,  manufacturer,  sup¬ 
porter,  tester  and  trainer);  and 

•  consists  of  various  classes  and  types  of 
M«&S  each  with  a  specific  purpose. 

This  chapter  provides  an  overview  of  how 
M&S  may  be  used  across  the  phases  of  ac¬ 
quisition  and  a  discussion  of  its  application 
to  specific  acquisition  related  activities. 

5.1  Modeling  and  Simulation  Across  the 
Acquisition  Life  Cycle 

The  DoD  Directive  5000.1  establishes  a  dis¬ 
ciplined  defense  acquisition  management 
approach  to  conducting  stable,  affordable 
acquisition  programs  that  meet  the  user’s 
needs.  This  management  framework  in¬ 
cludes: 


•  Evolving  broad  mission  needs  into  sys¬ 
tem  and  subsystem  requirements; 

•  Assessing  alternative  concepts  which 
eventually  develop  into  a  stable,  producible 
design  configuration;  and 

•  Establishing  initial  affordability  objec¬ 
tives  which  evolve  into  firm  unit  costs. 

The  DoD  Instruction  5000.2  describes  the 
management  policies  and  procedures  which 
are  to  be  applied  throughout  the  acquisition 
process  and  across  the  functional  disciplines. 
An  overview  of  the  process,  key  activities  and 
M&S  application  follows. 

It  is  recommended  that  readers  not  simply 
proceed  directly  to  the  discussion  regarding 
the  particular  phase  their  program  currently 
may  be  in.  Having  skipped  or  already  passed 
a  phase  in  development  does  not  negate  the 
need  for  conducting  those  past  development 
and  planning  activities  which  can  influence 
the  remainder  of  the  program.  Therefore, 
the  uses  of  models  and  simulations,  and  the 
planning  activities  discussed  in  conjunction 
with  earlier  phases  of  development,  (particu¬ 
larly  those  described  within  phase  0),  should 
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be  reviewed  regardless  of  the  current  phase 
of  the  program. 

5.1.1  PREMSO 

(Ref  DoDI  5000.2,  pg  3-2  through  3-6) 

5.1.1. 1  Focus:  To  determine  mission  need. 

5.1.1.2  Activities:  Mission  area  assessments 
(MAA)  and  mission  need  analyses  (MNA)  are 
conducted  to  examine  the  ability  of  existing 
systems  to  satisfy  mission  objectives  and  ana¬ 
lyze  capability  improvements  that  are  needed 
to  meet  deficiencies.  Assuming  that  the  defi¬ 
ciency  can  not  be  met  by  a  change  in  doctrine, 
tactics,  operational  concepts,  training  or  or¬ 
ganization  (called  “non  materiel”  solutions) 
a  mission  need  statement  (MNS)  will  be  writ¬ 
ten  describing  the  validated  threat,  deficien¬ 
cies,  constraints,  and  potential  alternatives  to 
overcome  the  deficiency. 

5.1.1.3  M&S:  As  with  many  other  analyses 
performed  in  support  of  a  system,  there  is 
no  single,  stand-alone  model  or  simulation 
to  conduct  the  MAA.  A  suite  of  models 
and  simulations,  along  with  supporting  data 
including  threat,  environment,  tactics,  etc. 
are  required. 

•  Engineering  level  models  of  new  de¬ 
signs  provide  system  and  subsystem  perfor¬ 
mance  to  support  higher  level  models. 

•  Engagement  and  mission/battle  level 
simulations  evaluate  the  effectiveness  of 
designs  in  an  operational  environment  and 
evaluate  the  consequences  of  different  en¬ 
gagement  tactics. 

•  Campaign/theater  level  models  exam¬ 
ine  the  outcomes  of  new  system  capabilities, 
technologies,  and  tactics  in  extended,  com¬ 
bined  force  conflicts. 


•  Human  interaction  in  simulations  may 
be  used  to  either  identify  the  tactics  for  use 
in  other  models  and  simulations,  or  to  ex¬ 
amine  the  operational  impacts  of  alterna¬ 
tive  tactical  schemes  or  concepts  of  opera¬ 
tion.  The  reader  is  reminded  of  the  discus¬ 
sion  in  Chapter  4  regarding  the  two  uses  of 
human-interactive  simulations:  determina¬ 
tion  of  decision-making  and  its  effects;  and 
definition  of  human-machine  interfaces. 
The  purpose  served  here  is  within  the  first 
category. 

•  Virtual  prototypes  also  demonstrate 
military  utility  of  new  tactics,  technologies 
and  systems. 

This  suite  of  models  and  simulations  allows 
for  analytical  evaluation  of  tactics  or  con¬ 
cepts  of  operation  changes  with  existing 
baseline  systems  prior  to  evaluation  of  new 
systems  in  accordance  with  DoDl  5000.2.' 
The  campaign/theater  level  models  and 
simulations,  used  in  conjunction  with  the 
results  of  the  lower  level  models,  will  de¬ 
velop  the  data  used  to  identify  warfighting 
needs  to  be  documented  in  the  MNS.  The 
engagement  and  mission  level  models  will 
identify  the  features  and  characteristics  that 
provide  the  required  capabilities  with  po¬ 
tential  to  satisfy  those  needs. 

Chapter  1  introduced  a  “risk  cycle”  and  dis¬ 
cussed  the  concept  of  “operational  risk”. 
The  M&S  tools  used  at  this  time  provide 
insight  into  that  risk  and  furthermore,  are 
used  to  identify  either  non  materiel  or  ma¬ 
teriel  approaches  to  mitigate  those  risks. 

5.1.2  Phase  0  -  Concept  Exploration  and 
Deflnition  (CED) 

(Ref  DoDI  5000.2,  pg  3-7  through  3-9) 
5.1.2.1  Focus:  To  define  and  evaluate  the 
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feasibility  of  alternative  concepts  and  assess 
the  relative  merits  of  each  concept. 

5.1.2.2  Activities:  During  the  CED  phase, 
materiel  alternatives  are  examined  and  a 
cost  and  operational  effectiveness  analysis 
(COEA)  is  performed  to  determine  the  rela¬ 
tive  cost  effectiveness  of  those  concepts. 
Operational  requirements  are  defined  and 
documented  in  an  operational  requirements 
document  (ORD)  and  concepts  are  defined 
at  the  system  level  resulting  in  a  draft  system 
specification.  The  critical  system  characteris¬ 
tics  and  operational  constraints  are  defined, 
and  the  initial  cost,  schedule  and  performance 
objectives  are  developed.  High  risk  areas  and 
risk  management  approaches  are  identified. 
Initial  manufacturing  and  logistics  support 
planning  is  begun.  Training  devices  are  also 
identified  in  the  ORD  produced  during  this 
phase.  By  the  end  of  this  phase  the  system 
threat  is  validated  and  an  affordability  assess¬ 
ment  of  the  proposed  new  system  is  conducted 
to  determine  if  the  proposed  new  system  fits 
within  the  Defense  Planning  Guidance  and 
long  range  investment  plans.^ 

5.1.2.3  M&S:  In  the  CED  phase,  many  of 
the  same  classes  and  types  of  models  and 
simulations  used  to  define  requirements  are 
again  employed  to  examine  the  capabilities 
of  specific  materiel  alternatives  at  the  engi¬ 
neering  through  campaign  analysis  levels. 

•  Engineering  level  models  and  simula¬ 
tions  of  proposed  technologies  and  new  de¬ 
signs  will  be  used  to  project  performance  and 
examine  performance  trade-offs.  Logistics 
support  models  will  be  used  in  defining  the 
overall  support  concept  and  operations  and 
support  costs.  Cost  models,  will  be  used  to 
determine  life  cycle  costs  for  use  in  the  pro¬ 
gram  cost  estimate,  for  evaluation  of  alterna¬ 
tives  in  the  COEA,  and  to  develop  a  prelimi¬ 
nary  design  to  unit  production  cost  objective. 


•  Engagement  and  mission/battle  mod¬ 
els  and  simulations  will  be  used  to  deter¬ 
mine  mission  effectiveness  measures  for  the 
proposed  alternatives,  again  in  support  of 
the  COEA  and  ORD  development. 

•  The  theater/campaign  level  models  and 
simulations  will  be  used  to  evaluate  the  pro¬ 
posed  systems  and  determine  their  impact 
on  the  outcome  of  conflicts.  These  results 
will  support  the  COEA  and  ORD,  and  be 
used  to  evaluate  how  well  the  proposed 
system(s)  meet  the  previously  identified 
need. 

•  Human  interactive  simulations  will 
continue  to  be  used  to  develop  and  exam¬ 
ine  tactics  and  decisions  within  the  above 
framework  of  constructive  models. 

•  Virtual  simulations  might  be  used  to 
evaluate  new  technologies,  system  concepts 
and  tactics  in  a  realistic  battle  environment. 
This  may  range  from  a  single  simulator  to 
multiple  simulators  of  a  new  system  linked 
to  other  friendly  and  enemy  system  simula¬ 
tors  on  a  synthetic  battlefield. 

It  is  in  this  phase,  that  initial  program  plan¬ 
ning  takes  place  and  the  key  program  docu¬ 
ments  which  set  the  stage  for  the  entire  pro¬ 
gram  are  developed.  Consistency  should  be 
maintained  among  all  of  the  acquisition 
management  documentation.  This  includes 
the  measures  of  effectiveness,  measures  of 
performance,  and  criteria  in  the  Operational 
Requirements  Document,  the  cost  and  op¬ 
erational  effectiveness  analysis,  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  the  ac¬ 
quisition  program  baseline.  Paraphrasing 
the  words  of  one  acquisition  manager,  “If 
you  cannot  show  how  you  intend  to  relate 
the  measures  of  effectiveness  (MOEs)  and 
measures  of  performance  (MOPs)  used 
across  the  COEA,  ORD,  acquisition  pro- 
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Relationship  of  Program  Documents,  Needs  and  Measures 

M&S  Establishes  anC  rAdintains  Consistency 


•  MOP 


•  THRESHOLDS  &  OBJECTIVES 
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TECHNICAL  SPECIFICATIONS 
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MEASURES  OF  EFFECTIVENESS 
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MEASURES  OF  PERFORMANCE 

•  THRESHOLDS  &  OBJECTIVES 


Figure  5-1 .  Relationship  of  Program  Documents,  Needs  and  Measures 


gram  baseline  (APB),  TEMP,  as  well  as  to 
the  MNS,  then  don’t  bother  showing  up  to 
the  DAB  planning  meeting.”  This  linkage 
is  depicted  in  Figure  5-1, 

The  TEMP,  first  prepared  during  this  phase, 
must  identify  M«feS  resources  which  will  be 
used  to  support  development  and  opera¬ 
tional  testing.^  The  program  office  should 
consider,  at  this  time,  how  models  and  simu¬ 
lations  will  be  used  across  all  of  the  func¬ 
tional  disciplines  as  the  appropriate  plans 
(e.g.,  integrated  logistics  support  plan,  sys¬ 


tems  engineering  management  plan  and 
manufacturing  plan)  are  developed. 

Modeling  and  simulation  is  a  powerful  tool 
to  assist  the  acquisition  manager  in  estab¬ 
lishing  and  maintaining  a  consistent  relation¬ 
ship  among  MOEs,  MOPs  and  program 
documentation.  The  activities  initiated  in 
this  phase  will  continue  or  be  repeated  with 
increasing  detail  and  specificity  as  the  sys¬ 
tem  design  matures.  The  program  office 
must  lay  the  groundwork  for  continuing  ap¬ 
plication  of  models  and  simulations  through- 
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out  the  life  cycle.  This  should  include  con¬ 
siderations  of  factors  such  as  use  and  reuse 
of  models  and  simulations,  integration  and 
interoperability  and  common  data  bases. 
Planning  for  model  and  simulation  devel¬ 
opment  should  also  address  future  compat¬ 
ibility  with  the  synthetic  battlefield  through 
DIS  communication  standards  and  the  even¬ 
tual  transition  or  application  of  developed 
models  and  simulations  to  the  training  en¬ 
vironment.  The  main  objective  is  to  allow 
the  program  to  later  build  upon  models  al¬ 
ready  developed,  thus  reducing  duplication 
and  providing  for  consistency  through  the 
phases  and  among  the  documents  and  ac¬ 
tivities  within  a  given  phase. 

The  use  of  models  and  simulations  can  sup¬ 
port  early  risk  management  activities  in  the 
acquisition  program.  The  M&S  tools  will 
identify  system  performance  and  effective¬ 
ness  levels  required  to  meet  the  specified 
threat.  Acquisition  managers  may  use  these 
results  to  aid  in  establishing  risk  levels  as¬ 
sociated  with  each  of  the  functional  areas 
(threat,  technology,  design  and  engineering, 
manufacturing,  support,  cost,  and  schedule) 
to  be  reported  within  the  risk  assessment  of 
the  integrated  program  summary.”*  The 
M&S  tools  can  also  support  evaluation  of 
system  or  technology  alternatives  which  may 
offer  reduced  risk.  From  a  program  plan¬ 
ning  point  of  view,  models  and  simulations 
can  serve  as  a  tool  to  assist  in  relating  the 
MOP,  MOE  and  integrating  program  plans, 
documentation,  and  functional  disciplines, 
fostering  Integrated  Product  and  Process 
Development  (IPPD)  at  the  start  of  the  pro¬ 
gram. 

5.1.3  Phase  I  -  Demonstration  and 
Validation  (DemVal) 

(Ref  DoDI  5000.2,  pg  3-13  through  3-15) 


5.1.3.1  Focus:  To  examine  multiple  design 
approaches  and  technologies  for  system  can¬ 
didates;  address  supportability,  manufactur¬ 
ing,  and  affordability;  and  establish  perfor¬ 
mance  objectives. 

5.1.3.2  Activities:  As  the  system  begins  to 
be  defined  at  greater  levels  of  detail,  draft 
development  specifications  are  produced; 
and  the  system  specification  is  approved  ini¬ 
tiating  configuration  management  of  the 
system.  Critical  design  characteristics  and 
performance  requirements  at  the  system 
level  are  refined  and  preliminary  require¬ 
ments  for  subsystems  are  developed.  A  new 
COEA  is  conducted  and  program  plans  and 
documentation  such  as  the  ORD,  TEMP, 
SEMP,  and  ILSP  and  acquisition  strategy  are 
updated.  Critical  technologies  are  demon¬ 
strated;  and  prototyping,  testing,  and  early 
operational  assessment  of  critical  systems, 
subsystems  and  components  are  conducted. 
Risk  areas  and  management  actions  will  be 
assessed.  Producibility  engineering  planning 
is  conducted  and  a  preliminary  manufactur¬ 
ing  plan  is  produced.  Logistics  support  ac¬ 
tivities  examine  factors  such  as  level  of  re¬ 
pair.  These  analyses  provide  a  basis  for  com¬ 
munication  between  logistics  and  design 
engineers  as  initial  design  trade-offs  are 
made. 

5.U  J  M&S:  Modeling  and  simulation  in  the 
Demonstration  and  Validation  (DemVal) 
phase  continues  to  support  and  extend  activi¬ 
ties  that  were  initially  conducted  in  CED. 
During  DemVal,  as  the  focus  of  develop¬ 
ment  starts  shifting  toward  design  of  sub¬ 
systems  and  components,  the  models  and 
simulations  take  on  better  definition.  Uses 
of  models  and  simulations  in  this  phase  in¬ 
clude: 

•  Engineering  level  models  and  simula¬ 
tions  of  proposed  system  and  subsystem  con- 
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cepts  at  increasing  levels  of  detail  will  be 
used  to  provide  a  better  estimate  of  perfor¬ 
mance  for  development  of  design  specifica¬ 
tions  and  for  use  in  other  models.  Hard- 
ware/software-in-the-loop  (HW/SWIL) 
simulations  will  be  used  to  evaluate 
brassboard  designs,  plan  prototype  tests  and 
to  identify  and  correct  problems  as  a  risk 
reduction  measure.  Logistics  support  mod¬ 
els  such  as  repair  level  analyses  allow  the 
logistics  and  design  community  to  investi¬ 
gate  the  sensitivities  of  reliability  and  main¬ 
tainability,  and  repair  level  in  concert  with 
design  trade-offs. 

Computer-aided  design  (CAD)  and  com¬ 
puter-aided  manufacturing  (CAM)  models 
support  both  design  and  producibility  plan¬ 
ning.  By  the  end  of  this  phase,  the  program 
office  should  require  factory  simulations  to 
support  HMD  and  subsequent  proposals  for 
rate  production  and  facilitization. 

Cost  models  will  begin  to  incorporate  engi¬ 
neering  cost  estimates  and  these  will  be  used 
to  determine  life  cycle  costs  in  the  program 
cost  estimate,  to  evaluate  alternatives  in  the 
COEA,  and  to  refine  the  design  to  unit  pro¬ 
duction  cost  estimate. 

•  Engagement  and  mission/battle  mod¬ 
els  and  simulations  will  again  be  used  to 
evaluate  mission  effectiveness  measures  for 
the  proposed  systems  in  support  of  the 
COEA  and  ORD.  They  may  also  be  used  to 
define  the  interoperability  requirements 
among  systems  and  the  impacts  of  the  pro¬ 
posed  system  on  existing  weapon  systems. 

•  The  theater/campaign  level  models  and 
simulations  will  be  used  to  evaluate  the  pro¬ 
posed  system’s  impact  on  the  outcome  of 
conflicts.  These  results  will  support  the 
COEA,  and  evaluate  how  well  the  system 
meets  the  requirements  stated  in  the  ORD. 


•  Human  interactive  simulations  will 
continue  to  examine  tactics  within  the  above 
framework  of  constructive  models  and  will 
begin  to  examine  the  human-machine  inter¬ 
face  in  virtual  simulations. 

•  Virtual  simulations  may  be  used  to 
evaluate  the  systems  and  subsystems  and  ex¬ 
amine  tactics  in  a  realistic  battle  environ¬ 
ment.  In  the  DemVal  phase,  such  simula¬ 
tions  can  employ  more  detailed  perfor¬ 
mance  models  and  actual  prototype  HW/ 
SWIL.  This  may  range  from  a  single  simu¬ 
lator  to  multiple  simulators  of  a  new  system 
linked  to  other  friendly  forces  and  engaging 
the  enemy  on  a  synthetic  battlefield.  These 
linkages  may  include  any  combination  of 
virtual  simulations,  live  simulations  (exer¬ 
cises),  or  constructive  models  of  systems  and 
forces.  The  virtual  prototypes  will  be  par¬ 
ticularly  useful  in  examining  human-ma¬ 
chine  interfaces  and  conducting  trade-offs 
without  building  actual  hardware. 

During  DemVal,  the  program  office  should 
be  taking  advantage  of  and  implementing 
the  plans  for  M&S  developed  during  CED. 
Simulations  at  various  levels  may  be  used 
in  support  of  the  early  operational  assess¬ 
ments,  particularly  when  little  hardware  is 
available  for  live  testing.  Simulations  can 
also  be  used  in  this  phase  to  assist  in  source 
selection.  Contractors  might  be  required  to 
bring  their  hardware  into  a  government 
simulation  facility  for  evaluation.  In  some 
instances,  contractors  might  be  provided  a 
copy  of  the  simulations  that  the  government 
intends  to  use  in  system  comparisons  dur¬ 
ing  source  selection. 

During  this  phase,  the  use  of  M«feS  will  con¬ 
tinue  to  help  the  acquisition  manager  to 
manage  risk.  The  early  use  of  simulations 
such  as  HW/SWIL  with  brassboard  equip¬ 
ment  can  allow  early  identification  and  so- 
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lution  of  technical  problems.  The  virtual  pro¬ 
totypes  with  a  single  common  data  base  which 
are  accessible  to  all  the  functional  disciplines 
will  facilitate  IPPD;  and  allow  better  design 
decisions  with  less  potential  for  having  over¬ 
looked  impacts  to  other  disciplines. 

5.1.4  Phase  II  -  Engineering  and  Manu¬ 
facturing  Development  (EMD) 

(Ref  DoDI  5000.2,  pg  3-20  through  3-22) 

5.1.4.1  Focus:  To  translate  the  most  prom¬ 
ising  design  approach  into  a  stable,  produc¬ 
ible  and  affordable  design. 

5.1.4.2  Activities:  The  EMD  phase  includes 
detailed  system,  subsystem  and  component 
design;  including  the  associated  manufactur¬ 
ing  and  support  processes.  Testing  is  con¬ 
ducted  to  verify  that  the  system  meets  speci¬ 
fication  requirements  and  satisfies  mission 
need  and  minimum  operational  perfor¬ 
mance  requirements.  Production  planning 
is  refined  and  logistics  support  activities  in¬ 
clude  provisioning.  Low  Rate  Initial  Produc¬ 
tion  (LRIP)  late  in  this  phase  is  conducted 
to  verify  producibility  and  production  costs. 
By  the  end  of  this  Phase,  the  system  is  de¬ 
fined  to  its  lowest  levels  (individual  parts). 
Initial  operational  test  and  evaluation 
(lOT&E)  and  live  fire  testing  is  completed 
prior  to  a  decision  for  full  rate  production. 

5.1.4.3  M&S:  A  major  focus  of  M&S  in 
EMD  is  at  the  engineering  level  models 
which  are  used  for  design,  engineering  trade¬ 
offs.  test  planning  and  support,  subsystem 
and  system  performance,  and  verification  of 
compliance  with  specifications.  Models  and 
simulations  also  support  COEA  and  ORD 
updates,  DT  and  OT«feE,  and  prepare  for 
production  and  deployment  of  the  system. 

Several  uses  of  models  and  simulations  in 


EMD  are  defined  below. 

•  Engineering  level  models  and  simula¬ 
tions  of  proposed  systems  and  subsystems 
will  be  used  for  detailed  design  and  assem¬ 
bly  of  subsystems,  components  and  piece 
parts.  Performance  requirements  will  be 
verified  using  a  combination  of  testing  and 
simulation. 

The  H'*  mulations  will  be  used  in 

a  mode,  ,  ,uodel  process  for  pre-test 
planning,  test  execution,  and  post-test  analy¬ 
sis.  Such  simulations  are  able  to  identify 
problems  in  actual  test  hardware  before  con¬ 
ducting  live  tests  (i.e.  live  simulations)  on 
the  range.  They  also  provide  for  parameter 
variation  studies,  and  augment  the  matrix 
of  test  conditions.  The  performance  esti¬ 
mates  from  simulations  during  this  phase 
along  with  live  simulation  (test)  data  pro¬ 
vide  input  for  models  and  simulations  at 
other  levels  or  of  other  classes. 

Logistics  support  models  will  examine  such 
factors  as  reliability,  availability,  maintain¬ 
ability,  transportability,  and  provisioning 
(spares,  support  equipment,  manpower). 
The  CAD/CAM  models  will  produce  de¬ 
signs  that  can  be  electronically  transmitted 
to  the  shop  floor  resulting  in  fewer  manu¬ 
facturing  errors.  Factory  simulations  are 
used  to  plan  facilities  and  equipment  and 
define  production  flows  to  meet  planned 
production  rates  in  support  of  both  design 
and  producibility  planning.  If  not  already 
accomplished,  the  program  office  should 
require  factory  simulations  to  support  pro¬ 
posals  to  substantiate  the  ability  to  achieve 
rate  production  and  identify  required 
facilitization. 

Cost  models  will  be  able  to  incorporate!  cost 
data  from  engineering  models  and  actual 
LRIP  hardware  for  the  program  cost  esti- 
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mates  and  COEA  updates. 

•  Engagement  and  mission/battle  mod¬ 
els  and  simulations  will  again  be  used  to 
evaluate  how  well  the  designs  allow  the  pro¬ 
posed  system  to  achieve  the  necessary  MOE. 

•  The  theater/campaign  level  models  and 
simulations  will  be  used  to  assess  the  pro¬ 
posed  system  and  determine  its  impact  on 
the  outcome  of  conflicts. 

•  Human  interactive  simulations  will 
continue  to  examine  tactics  within  the  above 
framework  of  constructive  models,  but  will 
more  likely  focus  on  continued  refinement 
of  human-machine  interfaces. 

•  Virtual  simulations  can  be  used  to 
evaluate  systems  performance  and  effective¬ 
ness.  A  virtual  prototype  can  be  used  to  sup¬ 
port  development  efforts  including  design, 
support  (e.g.  maintenance  walk  throughs), 
manufacturing  and  training.  Members  from 
every  functional  discipline  share  the  same 
electronic  representation  of  the  system  fa¬ 
cilitating  integrated  product  and  process 
development.  Weapon  system  trainers  be¬ 
ing  developed  should  take  maximum  advan¬ 
tage  of  the  models  and  simulations  used  in 
developing  the  system  itself.  As  these  train¬ 
ers  are  developed  and  made  available,  they 
may  be  used  for  training  test  crews,  and 
mission  rehearsal  for  live  simulations  (e.g. 
OT&E  planning). 

•  Live  simulations  may  take  the  form  of 
live  exercises,  or  instrumented  prototype 
tests,  including  lOT&E.  Managers  should 
insist  that  data  obtained  in  these  tests  are 
used  to  further  validate  the  models  and 
simulations. 

A  combination  of  engineering,  engagement, 
mission  and  campaign  simulations,  as  de¬ 


scribed  in  the  program  TEMP,  wilt  be  re¬ 
quired  to  augment  the  developmental  and 
operational  test  program  during  EMD.  Ear¬ 
lier  program  efforts  to  define  the  appropri¬ 
ate  models  and  simulations;  VV&A  them; 
and  determine  the  relationships  among 
MOE  and  MOP  are  critical  to  the  success¬ 
ful  application  of  M&S  to  support  or  aug¬ 
ment  the  test  program. 

At  the  end  of  EMD,  detailed  design  of  the 
system  including  definition  of  production 
and  support  processes  is  complete.  In  ac¬ 
cordance  with  M&S  planning  conducted 
beginning  in  the  CED  phase,  the  program 
office  should  be  prepared  to  maintain  those 
models  and  simulations  which  will  be  needed 
for  continued  support  of  the  weapon  system 
during  its  life  cycle.  The  program  manager 
(PM)  also  needs  to  consider  how  to  make 
representations  (models)  of  the  system 
available  to  others  outside  the  program  of¬ 
fice  that  may  have  a  need  to  use  them. 

Models  and  simulations  will  support  de¬ 
tailed  design  during  this  phase.  They  will 
continue  to  be  key  tools  for  IPPD,  and  will 
reduce  design  risk  by  allowing  all  of  the  func¬ 
tional  disciplines  to  work  from  the  same 
design  data  base.  A  reduced  number  of  en¬ 
gineering  change  proposals  (ECP)  will  be 
an  important  result  of  this  activity.  The  HW/ 
SWIL  simulations  will  result  in  significant 
risk  reduction  in  test  and  evaluation  through 
planning,  hardware  checkout  and  mission 
rehearsal.  Finally,  the  transition  to  produc¬ 
tion  will  take  place  with  reduced  risk  by  the 
electronic  transfer  of  digital  design  data  di¬ 
rectly  to  the  manufacturing  floor. 

5.1.5  Phase  III  -  Production  and 
Deployment 

(Ref  DoDI  5000.2,  pg  3-26  &  3-27) 
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5.1.5.1  Focus:  To  produce  and  deploy  the 
system. 

5.1.5.2  Activities:  The  objectives  of  this  phase 
are  to  establish  a  stable,  efficient  production 
and  support  base;  conduct  follow-on  opera¬ 
tional  test  and  evaluation  (FOT&E);  achieve 
operational  capability  that  satisfies  mission 
need;  identify,  verify,  and  incorporate  engi¬ 
neering  changes;  identify  operational  and/or 
support  problems;  and  identify  the  need  for 
major  upgrades  or  modifications  requiring  an 
MS  IV  review. 

5.1.5J  M&S:  In  the  production  and  de¬ 
ployment  phase,  models  and  simulations  can 
be  used  to  support  continued  testing,  verifi¬ 
cation  of  design  changes,  training  of  crew 
members  and  development  of  operational 
tactics.  Below  are  some  applications  for  this 
phase  of  the  program. 

•  Engineering  level  models  and  simula¬ 
tions  may  be  used  for  evaluation  and  verifi¬ 
cation  of  engineering  design  changes.  The 
HW/SWIL  simulations  will  continue  to  be 
used  in  a  model-test-model  process  for  pre¬ 
test  planning,  test  execution,  and  post-test 
analysis  in  support  of  further  development 
testing  and  FOT&E.  They  continue  to  be 
used  to  check  out  actual  test  hardware  and 
software  before  conducting  live  tests  (simu¬ 
lations)  on  the  range.  Simulations  may  also 
be  used  in  production  lot  sampling  or  pro¬ 
duction  acceptance  as  a  less  expensive  al¬ 
ternative  to  live  testing.  Logistics  support 
models  will  incorporate  actual  field  data  to 
determine  system  readiness. 

Factory  simulations  can  evaluate  the  effects 
of  design  and  manufacturing  process 
changes  on  production.  They  can  also  pro¬ 
vide  information  on  whether  current  facilities 
and  resources  can  accommodate  changes  in 
production  rates  or  the  manufacturer’s  busi¬ 


ness  base.  The  support  community  may  also 
choose  to  use  such  simulations  in  depot  plan¬ 
ning  for  weapon  system  maintenance. 

Cost  models  will  have  the  ability  to  use  ac¬ 
tual  production  cost  data  and  preliminary 
O&S  costs  in  maintaining  oversight  of  pro¬ 
gram  costs. 

•  Engagement,  mission/battle  and  the¬ 
ater/campaign  level  models  and  simulations 
may  continue  to  be  used  to  evaluate  opera¬ 
tional  consequences  of  system  performance 
changes  or  changes  in  threat.  In  the  inter¬ 
active  mode,  these  simulations  will  be  used 
as  wargaming  simulations  to  train  battle 
staffs  on  new  tactics  and  concepts  of  opera¬ 
tion. 

•  Virtual  simulations,  such  as  weapon 
system  trainers,  can  be  used  for  training  op¬ 
erational  crews  and  mission  rehearsal  for 
FOT&E.  They  can  also  be  used  along  with 
other  simulations  of  friendly  forces  to  en¬ 
gage  enemy  forces  on  a  synthetic  battlefield. 

•  Live  simulations  will  include  live  exer¬ 
cises,  or  instrumented  prototype  tests,  such 
as  FOT&E. 

With  the  system  being  produced  and  de¬ 
ployed,  real  data  are  becoming  available  for 
continued  validation  of  the  models  and 
simulations  used  within  the  program.  This 
is  an  opportunity  to  improve  the  models  for 
future  use  either  in  the  current  program  or 
another  program — it  is  useless  to  place  the 
data  on  the  shelf  and  forget  about  it! 

At  this  point  in  the  program,  the  program 
office  is  implementing  the  plans  for  life  cycle 
maintenance  of  the  models  and  simulations. 
This  is  particularly  important  to  keep  in 
mind  for  weapon  system  post-deployment 
support  activity. 
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Within  this  phase,  the  use  of  modeling  and 
simulation  will  continue  to  reduce  program 
risk.  On  the  acquisition  side,  the  electronic 
transmittal  of  design  data  to  the  shop  floor 
will  result  in  fewer  manufacturing  errors  and 
less  rework.  The  factory  should  be  able  to 
meet  production  requirements  based  upon 
planning:  using  the  factory  simulations. 

From  the  operational  perspective,  the 
weapon  system  operators  will  have  available 
a  training  system  which  accurately  simulates 
the  weapon  system  and  its  weapons.  It  will 
be  able  to  use  models  and  simulations  re¬ 
flecting  the  characteristics  of  its  weapons 
and  enemy  threats,  and  even  employ  the 
appropriate  data  protocols  so  it  may  inter¬ 
act  with  other  live  or  simulated  entities 
within  a  synthetic  battlefield  environment. 

5.1.6  Phase  IV  •  Operations  and  Support 
(O&S) 

(Ref  DoDI  5000.2,  pg  3-28  through  3-30) 

5.1.6.1  Focus:  Insure  system  continues  to 
meet  mission  needs  and  identify  shortcom¬ 
ings  and  deficiencies. 

5. 1.6.2  Activities:  In-service  engineering 
support,  implementation  of  design  changes 
and  service  life  extension  programs  are  an 
ongoing  activity.  A  change  in  threat,  a  defi¬ 
ciency  in  capability  or  the  opportunity  to 
reduce  the  cost  of  ownership  (such  as  via 
new  technology)  may  result  in  a  decision  to 
seek  a  major  modification  approval,  mile¬ 
stone  IV,  for  which  a  COEA  will  be  pre¬ 
pared. 

5.1.6  J  M&S:  In  the  O&S  phase,  models  and 
simulations  can  be  used  to  support  contin¬ 
ued  testing,  verification  of  design  changes, 
training  of  crew  members,  mission  rehearsal 
and  development  of  operational  tactics. 


More  importantly,  M&S  will  be  used  in  sup¬ 
port  of  decisions  to  initiate  major  modifica¬ 
tions  of  the  system  and  to  identify  deficien¬ 
cies. 

•  Engineering  level  models  and  simula¬ 
tions  will  continue  to  be  used  for  evaluation 
and  verification  of  engineering  changes.  The 
HW/SWIL  simulations  will  be  used  to 
“check  out”  actual  test  hardware  and  soft¬ 
ware  before  conducting  live  tests  (simula¬ 
tions)  on  the  range. 

For  some  systems  there  is  a  potential  for 
performance  to  degrade  as  a  result  of  long 
term  storage.  Shelf-life  evaluations  of  sys¬ 
tems  or  components  can  be  conducted  via 
simulation,  in  lieu  of  live  tests.  In  this  case, 
engineering  level  HW/SWIL  simulations  can 
evaluate  performance  of  articles  after  ex¬ 
tended  storage.  The  operational  impacts,  or 
potential  changes  in  tactics  to  accommodate 
performance  degradations,  can  then  be 
evaluated  in  the  higher  level  models. 

Logistics  support  models  will  continue  to  use 
actual  field  data  to  determine  system  .eadi- 
ness.  The  support  community  may  choose 
to  use  factory  simulations  to  support  depot 
planning  for  weapon  system  maintenance, 
particularly  useful  when  major  changes  in 
workload  arc  anticipated. 

Cost  models  will  have  the  ability  to  use  ac¬ 
tual  production  and  O&S  cost  data  to  main¬ 
tain  oversight  of  program  costs  and  support 
a  COEA  for  major  modification. 

•  Engagement,  mission/battle  and  the¬ 

ater/campaign  level  models  and  simulations 
may  continue  to  be  used  to  evaluate  opera¬ 
tional  consequences  of  system  performance 
changes  or  changes  in  threat.  In  the  inter¬ 
active  mode,  these  simul? '  ^  will  be  used 

as  wargaming  simulatic  rain  battle 
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staffs  on  tactics  and  concepts  of  operation. 
More  importantly,  these  models  and  simu¬ 
lations  may  be  used  in  identifying  deficien¬ 
cies  because  of  threat  changes,  or  may  den; 
onstrate  the  military  utility  of  new  technolo¬ 
gies  which  can  lead  to  a  milestone  IV  deci¬ 
sion  for  major  modification  approval. 

•  Virtual  simulations  will  continue  to  be 
used  for  training  operational  crews  and  mis¬ 
sion  rehearsal.  They  may  also  be  used  to  ex¬ 
amine  the  potential  of  new  technology  ap¬ 
plications  for  system  improvement  or  exam¬ 
ine  the  impact  of  threat  changes  within  a 
synthetic  battlefield  environment  with  other 
systems. 

•  Live  simulations  will  continue  to  in¬ 
clude  live  exercises,  or  instrumented  tests 
of  the  system.  With  the  system  in  use  within 
its  operating  environment,  either  in  train¬ 
ing  or  live  exercises,  real  data  are  available 
for  continued  validation  of  the  models  and 
simulations  used  within  the  program. 

Nearly  every  weapon  system  today  employs 
computer  software.  Software  updates  to 
improve  capability  or  to  counter  a  new  threat 
are  an  ongoing  activity  for  the  life  of  the 
system.  Simulations  are  the  primary  method 
used  to  test  and  verify  any  changes  made  in 
the  system  software.  This  fact  emphasizes 
the  importance  of  a  simulation  plan  that 
addresses  the  use  and  maintenance  of  mod¬ 
els  and  simulations  throughout  the  system 
life  cycle. 

The  “risk  cycle”  introduced  in  Chapter  1  has 
come  full  circle.  The  models  and  simulations 
will  be  used  to  reduce  risk  in  several  areas. 
They  will  be  employed  to  replicate  problems 
or  failures  encountered  in  operations  and 
verify  solutions  to  maintain  mission  capa¬ 
bility.  In  those  instances  where  performance 
has  degraded  over  time,  for  whatever  rea¬ 


son,  models  and  simulations  will  quantify 
performance  impacts  and  evaluate  changes 
in  tactics  or  employment  to  accommodate 
them.  Lastly,  the  suite  of  models  and  simu¬ 
lations  will  identify  capability  deficiencies 
because  of  threat  changes  and  evaluate  non 
materiel  and  materiel  approaches  to  miti¬ 
gate  the  operational  risk  caused  by  those 
deficiencies. 

5.1.7  Summary 

As  the  above  discussion  shows,  M&S  will 
be  used  by  a  program  from  the  earliest  stages 
through  operations  and  support.  Many  of 
the  same  models,  or  types  of  models  will  be 
used  repeatedly.  A  significant  amount  of 
program  planning  takes  place  during  phase  0, 
CED.  It  is  in  this  early  planning  stage  that 
M&S  should  be  identified  and  woven  into 
the  appropriate  program  and  functional  dis¬ 
cipline  plans. 

5.2  Modeling  and  Simulation  in  Support 
of  Acquisition  Activities 

The  remainder  of  the  chapter  describes  how 
M&S  supports  key  acquisition  functions  as 
they  span  the  phases  of  the  process.  These 
funct  iOns  are;  requirements  definition,  pro¬ 
gram  management,  design  and  engineering, 
manufacturing,  test  and  evaluation,  logistics 
support  and  training.  For  these  functional 
areas,  the  reader  will  find  a  template  in 
Appendix  G  showing  key  activities  con¬ 
ducted  and  the  types  of  models  and  simula¬ 
tions  that  might  be  employed.  Those  tem¬ 
plates  should  serve  as  examples  to  stimu¬ 
late  PMs  as  they  perform  the  detailed  plan¬ 
ning  for  their  own  programs.  In  applying 
M&S  to  activities  described  in  the  remain¬ 
der  of  this  chapter  to  their  own  programs, 
PMs  should  look  for  opportunities  in  two 
areas: 
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•  How  the  ,  )gram  can  use  the  M&S 
tools  across  phases  of  the  acquisition  pro¬ 
cess;  and 

•  How  the  program  might  make  use  of 
M&S  to  integrate  activities  across  functional 
boundaries. 

5.2.1  Requirements  Deflnition 

Chapter  2  provided  a  detailed  discussion  of 
the  requirements  development  process.  In 
this  section,  we  will  describe  the  types  of 
models  and  simulations  that  can  be  used  in 
the  process  of  developing  requirements 
documents  (MNS,  ORD,  specifications). 
The  reader  is  referred  to  Figure  5-2  for  the 
following  discussion. 

As  with  most  other  analyses  conducted  dur¬ 
ing  the  acquisition  process,  a  complimen¬ 
tary  suite  of  models  and  simulations  is  likely 
to  be  used,  ranging  from  engineering  per¬ 
formance  to  theater/campaign  levels. 

The  input  data  to  the  analysis  process  in¬ 
cludes  ground  rules  such  as: 

•  Defense  Intelligence  Agency  (DIA) 
threat  estimates  along  with  scenarios  and 
missions  derived  from  the  Defense  Planning 
Guidance  (DPG); 

•  Environmental  data  including  weather, 
terrain,  ocean  environment,  countermea¬ 
sures,  etc.; 

•  A  selection  of  operational  concepts  and 
tactics,  which  allow  for  evaluation  of  poten¬ 
tial  non-material  solutions  as  required  by 
DoDI  5000.2; 

•  System  options  to  include  existing,  up¬ 
grades  or  new  systems;  and 


•  New  technologies  that  may  be  available 
through  DoD’s  science  and  technology  pro¬ 
grams,  advanced  technology  demonstrations 
or  industry. 

These  data  address  a  variety  of  scenarios, 
systems  and  tactics  and  will  be  used  in  analy¬ 
ses  conducted  at  each  level  in  the  M&S  hi¬ 
erarchy  described  in  Chapter  4.  Using  the 
engineering  level  of  models,  analyses  pro¬ 
vide  performance  estimates  for  existing  and 
improved  capability  systems  taking  into  ac¬ 
count  the  emerging  technology  opportuni¬ 
ties.  The  performance  and  design  trade-offs 
of  system  and  subsystem  design  concepts  and 
technologies  are  evaluated  at  this  level. 
These  system/subsystem  performance  capa¬ 
bilities  are  evaluated  within  the  engagement 
and  mission/battle  level  models  and  simu¬ 
lations  to  determine  system  effectiveness 
(e.g.  probability  of  kill,  losses,  survivability, 
vulnerability)  and  mission  effectiveness  (e.g. 
loss  exchange  ratios,  probability  of  engage¬ 
ment)  in  a  limited  engagement  or  mission. 
These  capabilities  support  campaign  level 
models  to  examine  effects  of  force  mix,  tac¬ 
tics  or  new  capabilities  on  outcomes,  typi¬ 
cally  in  terms  of  force  exchange  ratios,  draw 
downs  or  troop  movements. 

The  analyses  are  repeated  for  a  variety  of 
operational  concepts  and  each  of  the  sys¬ 
tem  options  under  consideration.  The  en¬ 
gagement,  mission  and  campaign  models 
may  be  run  iteratively  to  provide  statistical 
significance  to  the  outcomes.  Material  ca¬ 
pability  needs  are  identified  and  docu¬ 
mented  in  a  MNS. 

The  engineering  models  in  conjunction  with 
the  engagement  and  mission/battle  level 
models  also  provide  the  basis  for  the  descrip¬ 
tion  of  broad  capabilities  and  technology  de¬ 
velopments  which  should  be  studied  in  CED. 
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Models  and  Simulations  in  Requirements  Definition 


Figure  5-2.  Models  and  Simulations  in  Requirements  Definition 
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ENGINEERING 

(SYSTEM/SUtSYSTEM/COMPONENT  PERFORMANCE) 


The  ORD  will  be  developed  during  the  CED 
phase.  The  ORD  defines  thresholds  and  ob¬ 
jectives  in  terms  of  operational  effectiveness 
measures,  system  performance  measures  and 
critical  system  characteristics.  The  ORD  will 
be  updated  during  DemVal  with  refined  and 
more  detailed  capabilities  and  characteristics. 
It  is  likely  that  mission/battle  and  engagement 
models,  in  conjunction  with  engineering  mod¬ 
els,  will  be  used  to  develop  the  effectiveness 
and  performance  measures  for  the  ORD. 

Technical  specifications  similarly  evolve.  A 
draft  system  level  specification  will  be  de¬ 
veloped  during  CED;  development  specifi¬ 
cations  will  be  written  during  DemVal;  and 
product,  process  and  material  specifications 
during  EMD.  Engineering  level  M&S  (e.g. 
design,  support,  manufacturing  and  HW/ 
SWIL)  typically  support  the  development  of 
these  requirements  specifications. 

There  is  not  a  simple  one-to-one  mapping 
between  a  particular  level  of  M&S  and  a 
particular  requirements  document.  Rather, 
as  was  discussed  in  Chapter  4,  a  combina¬ 
tion  of  M&S  (levels  and  classes)  will  likely 
be  needed  to  generate  the  various  measures 
and  insure  consistency  of  those  measures 
across  the  program  documents. 

5.2.2  Program  Management 

The  PM  is  faced  with  balancing  cost,  sched¬ 
ule  and  performance  objectives  throughout 
the  program.  Much  of  the  current  empha¬ 
sis  in  M&S  is  on  the  performance  or  mili¬ 
tary  utility  arena,  as  has  been  the  focus  of 
much  of  this  guidebook.  This  next  section 
will  touch  upon  some  of  the  management 
tools  that  are  in  existence. 

5.2.2.1  Program  Management  Tools 

Many  models  used  for  program  management 


are  actually  data  base  references,  or  knowl¬ 
edge-based  tools.  Examples  of  two  such  man¬ 
agement  tools  are  the  Navy’s  Program 
Manager’s  Work  Station  (PMWS)'  and  the 
Air  Force  Acquisition  Model  ( AFAM)'’.  These 
will  be  discussed,  followed  by  some  guidelines 
on  the  use  of  cost  models. 

Program  Manager’s  Work  Station  (PMWSjis, 
a  series  of  interrelated  software  tools  de¬ 
signed  to  provide  acquisition  information 
(primarily  engineering  process  oriented)  to 
the  user.  Three  of  the  modules  may  be  of 
particular  use  to  the  program  office: 

•  Know-how:  an  automated  handbook 
system  that  is  reported  to  reduce  search  time 
in  handbooks  and  manuals  such  as  the  DoD- 
5000  series.  Best  Practices  (NAVSO-P-6071) 
and  ISO  9000. 

•  Technical  Risk  Identification  and  Miti¬ 
gation  System  (TRIMS):  based  on  the  Best 
Practices  (NAVSO-P-6071)  it  allows  the  user 
to  develop  risk  metrics  and  status. 

•  Best  Manufacturing  Practices  data 
base:  Contains  over  1800  abstracts  from 
companies  documenting  best  practices. 

Air  Force  Acquisition  Model  (AFAM)  is  an 
acquisition  process  model  designed  to  facili¬ 
tate  the  process  for  major  and  non-major 
acquisition  programs.  This  personal  com¬ 
puter  based  model  addresses  activities 
across  the  acquisition  phases. 

•  Acquisition  Guides  and  Insights:  Pro¬ 
cedures  which  cover  2200  acquisition  tasks 
based  on  the  DoD  5000  series  and  best  man¬ 
agement  practices  and  lessons  learned. 

•  Acquisition  Management  Tools:  Pro¬ 
vides  guidance,  task  relationships,  timelines 
and  expertise  from  actual  program  experi- 
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ence  which  one  should  tailor  before  using 
in  a  specific  program. 

The  PMWS  and  AFAM  are  but  two  ex¬ 
amples  of  program  management  tools  which 
provide  PMs  with  a  disciplined  thought  pro¬ 
cess.  They  should  serve  to  prompt  PMs  re¬ 
garding  activities  that  should  be  performed 
and  application  of  best  practices. 

S.1.1.2  Cost  Models 

Program  managers  develop  cost  estimates 
during  the  acquisition  process  for  two  pur¬ 
poses: 

•  Program  life  cycle  cost  estimates;  and 

•  Cost  estimates  for  alternatives  evalua¬ 
tion  in  the  COEA. 

Two  separate  cost  estimates  are  required 
from  the  DoD  component  in  support  of 
milestone  I  and  subsequent  reviews.  One  of 
these  estimates  will  be  prepared  by  the  pro¬ 
gram  office  and  the  other  by  a  separate  or¬ 
ganization  that  does  not  report  through  the 
acquisition  chain. ^  Additionally,  the  OSD 
Cost  Analysis  Improvement  Group  (CAIG) 
will  develop  an  independent  DoD  estimate 
and  prepare  a  report  to  the  Under  Secre¬ 
tary  of  Defense  for  Acquisition  and  Tech¬ 
nology  (USD(A&T))  for  ACAT  ID  pro¬ 
grams,  and  to  the  DoD  Component  Acqui¬ 
sition  Executive  for  ACAT  IC  programs. 

The  second  use  of  cost  estimates  is  in  the 
preparation  of  the  COEA  to  support  mile¬ 
stone  decisions  beginning  with  milestone  I. 
The  COEA  is  prepared  by  an  independent 
activity  within  the  component,  and  should 
aid  decision-makers  in  judging  which,  if  any, 
of  the  proposed  alternatives  to  the  current 
program  offer  sufficient  military  benefit  to 
be  worth  the  cost.® 


We  will  not  address  specific  cost  models  used 
throughout  DoD  in  this  guidebook,  how¬ 
ever,  some  general  features  of  a  cost  model 
might  include: 

•  Cost  estimating  relationships 

•  Statistics  package 

•  Ability  to  address  various  cost  estimat¬ 
ing  methodologies 

•  Learning  curve  calculations 

•  Risk  analysis 

•  Sensitivity  analyses 

•  System  Work  Breakdown  Structure 
(WBS) 

•  Multiple  appropriations  (R&D,  Ap¬ 
propriations,  O&S) 

•  Time-phasing  of  costs 

•  Overhead  rates 

•  Inflation  indices. 

The  above  features  are  contained  in  the 
Automated  Cost  Estimating-Integrated 
Tools  (ACE-IT),'^  which  is  a  framework 
within  which  the  analyst  can  develop  a  cost 
model.  These  features  are  shown  only  as  an 
illustration  of  what  might  be  addressed  in  a 
cost  model,  and  are  not  necessarily  all-in¬ 
clusive  nor  must  any  particular  model  con¬ 
tain  all  those  features. 

The  Office  of  the  Secretary  of  Defense 
(OSD)  Cost  Analysis  Guide  provides  some 
guidelines  regarding  the  characteristics  of  a 
good  model  for  0«&S  costing  which,  with 
tailoring,  might  be  useful  for  any  model  ap¬ 
plication.'” 

•  Consistency  in  cost  element  structure: 
The  basic  cost  structure  should  not  change 
as  a  system  passes  through  the  acquisition 
phases.  However,  the  basic  elements  and 
their  sub-elements  should  be  expanded  to 
capture  greater  levels  of  detail. 

•  Consistency  in  data  elements:  Data  el- 
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ements  of  the  proposed  system  should  be 
consistent  with  those  of  operational  systems 
for  which  actual  data  exists.  This  allows  the 
costs  and  cost  driving  parameters  of  the  ref¬ 
erence  and  proposed  system  to  be  com¬ 
pared. 

•  Flexibility  in  estimating  techniques: 
The  estimating  techniques  should  be  al¬ 
lowed  to  vary  as  a  program  progresses 
through  the  various  acquisition  phases. 

•  Simplicity:  Complexity  is  not  desirable 
in  an  O&S  cost  model.  Models  should  be 
structured  in  a  way  that  allows  them  to  ac¬ 
commodate  more  detailed  information  as  a 
program  progresses  through  the  life  cycle. 

•  Usefulness  to  the  Design  process: 
While  estimating  costs  for  a  CAIG  review 
is  an  important  function,  a  model’s  applica¬ 
bility  to  day-to-day  program  office  and  con¬ 
tractor  decision  making  is  equally  important. 

•  Completeness:  The  model  should  cap¬ 
ture  all  significant  costs  that  will  be  incurred 
by  the  weapon  system  over  its  useful  life. 

•  Validity:  The  model  should  provide 
sound,  reproducible  results  for  its  intended 
application. 

The  PM  should  recognize  that  in  actual  prac¬ 
tice,  cost  estimating  is  a  melding  of  art  and 
science.  There  is  no  “one  model  fits  all”,  but 
rather  typically  a  custom  model  for  each 
program,  relying  on  various  cost  methodolo¬ 
gies  or  historical  data  bases  to  address  dif¬ 
ferent  elements  of  the  system.  As  with  any 
other  M&S  efforts,  an  experienced  analyst 
is  key  to  obtaining  credible  results. 

The  Cost  Analysis  Requirements  Document 
(CARD)  describes  the  system  and  salient  fea¬ 
tures  of  the  program  which  will  be  used  to 


develop  life  cycle  cost  estimates."  It  provides 
a  description  of  the  system  and  its  key  charac¬ 
teristics  (weight,  size,  payload,  speed,  power, 
etc.)  for  each  WBS  element.  The  CARD  ad¬ 
dresses  the  operational  concept,  risk,  quanti¬ 
ties,  manpower,  system  usage  rates,  schedules, 
acquisition  strategy,  development  plans  and 
facilities  requirements.  Since  the  CARD  ad¬ 
dresses  all  the  key  cost  elements  of  the  sys¬ 
tem,  it  provides  the  basis  for  cost  estimating 
and  the  use  of  cost  models. 

A  study  being  conducted  by  the  Army  Cost 
and  Economic  Analysis  Center  will  provide 
the  results  of  a  survey  on  cost  models  used 
throughout  DoD,  and  may  be  of  interest  to 
Program  Managers.'^ 

5.23  Design  and  Engineering 

The  use  of  M&S  is  most  prevalent  in  this 
functional  discipline.  A  visible  example  is 
the  Boeing  777  aircraft.  This  is  the  first  air¬ 
plane  designed  solely  by  computer  which  was 
accomplished  largely  via  the  CATIA  (Com¬ 
puter  Aided  Three  Dimensional  Interactive 
Application)'^  system.  Significant  accom¬ 
plishments  of  this  effort  included: 

•  “Paperless”  design  -  the  blueprints  re¬ 
sided  in  the  computer; 

•  Design/build  teams  shared  the  same 
design  information  contained  in  the  com¬ 
puter  data  base.  2200  networked  work  sta¬ 
tions  allowed  all  of  the  functional  disciplines 
(e.g.  Users,  Design,  Manufacturing,  Main¬ 
tenance)  to  communicate  based  on  a  com¬ 
mon  frame  of  reference; 

•  The  3-D  “virtual  airplane”  allowed  en¬ 
gineers  to  make  design  changes  and  visual¬ 
ize  the  results,  such  as  component  interfer¬ 
ences;  and 
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•  Design  data  transferred  electronically 
to  the  manufacturing  floor. ''' 

The  results  of  this  approach  included  more 
than  a  50  percent  reduction  in  change  error 
and  rework  in  manufacturing.’^ 

Modeling  and  simulation  pervades  the  vari¬ 
ous  specialty  disciplines  involved  with  de¬ 
sign — ranging  from  finite  element  analysis 
for  structural  design,  to  computational  fluid 
dynamics  for  aerodynamics  or  hydrodynam¬ 
ics.  For  human  factors,  anthropometric 
models  such  as  “Jack”  can  be  used  to  exam¬ 
ine  the  ability  of  a  crew  member  to  operate 
controls,  repair  equipment  or  fit  within  crew 
compartments.  What  these  models  and 


simulations  offer  is  the  ability  to  modify 
designs,  analyze  the  effects  and  refine  the 
design  repeatedly  prior  to  building  a  single 
hardware  prototype. 

In  the  future,  with  the  integration  of  design 
and  performance  simulation  models,  one 
can  achieve  a  “Simulation  Based  Design”'® 
in  which  3-D  virtual  prototypes,  properly 
representing  both  design  and  performance, 
function  realistically  in  a  virtual  environ¬ 
ment  and  replace  actual  hardware  mockups. 
Figure  5-3  depicts  the  process  whereby  all 
of  the  functional  disciplines  will  use  the  same 
virtual  prototype  to  support  activities  across 
the  system  life  cycle — from  operational  re¬ 
quirements  generation  through  engineering. 


Figure  5-3.  Simulation  Based  Design 
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construction,  testing,  training  and  operations 
and  logistics  support.'^  The  reader  will  see 
another  example  of  the  use  of  virtual 
prototyping  in  vehicle  design  in  Chapter  8. 

5.2.4  Manufacturing 

Producibility  is  intimately  linked  with  prod¬ 
uct  design — shape,  features,  materials,  etc. 
The  use  of  computer  models  to  simulate 
manufacturing  processes  such  as  metal 
forming,  machining  and  casting  allows  one 
to  evaluate  the  ability  to  produce  a  design 
before  actually  “bending  metal”.  The  use 
of  CAD/CAM  models  allows  the  design  and 
manufacturing  communities  to  converge  on 
a  producible  design  that  meets  the  require¬ 
ment.  Using  the  same  models  and  simula¬ 
tions  for  design  and  manufacturing,  com¬ 
bined  with  the  transfer  of  digital  design  data 
bases  directly  to  the  manufacturing  floor, 
reduces  errors,  rework  and  hence,  produc¬ 
tion  risk. 

In  addition  to  having  a  producible  design, 
the  program  office  must  be  assured  that  the 
necessary  capability/capacity  is  available  to 
meet  planned  production  rates. 

In  the  CED  phase,  production  planning  be¬ 
gins  with  an  industrial  base  analysis.  Con¬ 
siderations  include  the  investments  neces¬ 
sary  for  industrial  capabilities  to  provide  and 
sustain  production,  tooling  and  facilities."* 
During  the  DemVal  phase,  an  initial  manu¬ 
facturing  plan  is  developed  to  portray  the 
facilities,  tooling  and  personnel  resources 
required  for  production.”  This  plan  is  up¬ 
dated  during  the  EMD  phase  based  upon 
the  planned  detailed  manufacturing  opera¬ 
tions.  In  production  readiness  re^'iews,  con¬ 
ducted  during  EMD,  the  program  manage¬ 
ment  office  (PMO)  will  evaluate  the  capac¬ 
ity  of  the  production  facility  to  meet  the  re¬ 
quired  production  rates.  The  PMO  will  also 


evaluated  the  contractor’s  production  plan¬ 
ning,  including  manufacturing  methods  and 
processes,  facilities,  equipment  and  tooling, 
and  plant  layout.^** 

Factory  simulations  are  used  to  aid  in  this 
cycle  of  production  planning  which  can  sup¬ 
port  the  activities  mentioned  above.  These 
simulation  tools  can  address  production  pro¬ 
cesses,  factory  process  flow,  statistical  varia¬ 
tion  in  manufacturing  operations,  equip¬ 
ment,  plant  layout  and  manpower  require¬ 
ments  to  meet  production  demands.  Mili¬ 
tary  and  commercial  programs  are  turning 
to  such  tools  to  improve  efficiency  or  deter¬ 
mine  facilitization  requirements.  These  tools 
may  b^*  used  for  planning  a  new  production 
activity,  or  to  examine  changes  to  an  exist¬ 
ing  program.  An  example  follows  showing 
the  use  of  simulation  to  plan  changes  in  the 
periodic  maintenance  of  C-141  aircraft.^’ 

In  this  case,  the  periodic  depot  maintenance 
(PDM)  of  the  C-141  aircraft  fleet  was  im¬ 
pacted  when  two  structural  problems  were 
discovered:  wing  and  center  wing  box  cracks. 
Repair  of  the  wing  cracks  and  replacement 
of  the  center  wing  box  needed  to  be  incor¬ 
porated  into  the  ongoing  PDM  of  the  air¬ 
craft.  Furthermore,  replacement  of  the  cen¬ 
ter  wing  box  was  a  new  process  for  the  de¬ 
pot — it  had  only  been  done  once  on  a  pro¬ 
totype  aircraft  at  a  contractor’s  facility.  The 
SLAM  II  simulation  language  was  used  to 
simulate  the  ongoing  PDM  along  with  the 
introduction  of  the  wing  repair  and  center 
box  replacement.  A  sample  of  the  results  of 
this  simulation  include: 

•  An  achievable  schedule  for  wing  box 
replacement,  but  a  shortfall  for  wing  crack 
repair; 

•  Bottleneck  locations;  and 
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•  The  preference  to  reallocate  rather 
than  purchase  additional  inspection  equip¬ 
ment. 

Commercially  available  and  industry  owned 
factory  simulations  are  in  use  by  m.any 
weapon  .sy.stem  contractors  or  maintenance 
depots  today.  Factory  simulations  such  as 
“Witness”--  are  now  being  regularly  used  to 
support  aircraft,  missiles,  and  electronics 
production,  and  depot  activities.  A  listing 
of  commercially  available  manufacturing 
related  simulation  programs  can  be  found 
in  reference  23.-’ 

Factory  simulations  can  be  used  for  the  fol¬ 
lowing: 

•  Develop  an  assembly  strategy; 

•  Graphically  model  the  assembly  se¬ 
quence; 

•  Develop  and  validate  work  sequences; 

•  Develop  and  validate  manufacturing 
process  plans; 

•  Model  the  factory  floor,  including  fa¬ 
cilities  and  equipment; 

•  Identify  what  is  achievable  in  terms  of 
cost  and  schedule; 

•  Identify  bottlenecks; 

•  Compare  different  manufacturing 
strategies;  and 

•  Identify  impacts  of  engineering 
changes,  new  materials,  machines  or  pro¬ 
cesses. ’■* 

/  .11  of  these  factors  arc  important  in  deter¬ 
mining  the  robustness  of  production  plan¬ 


ning  in  proposal  evaluation,  or  eventually, 
readiness  for  productit)n.  I'heir  use  by  the 
contractors,  beginning  no  later  than  the 
DemVal  phase  can  ensure  the  program  of¬ 
fice  that  proper  production  planning  has 
been  accomplished. 

The  Air  Force  has  initiated  a  policy  regard¬ 
ing  the  use  of  factory  simulations  in  support 
of  depot  upgrades.  In  November  1992,  a 
policy  letter  was  written  establishing  the  re¬ 
quirement  that  all  depot  maintenance  .spon¬ 
sored  military  construction  projects  or 
equipment  projects  greater  than  $0.5M  shall 
be  modeled  as  a  prerequisite  to  funding.- 

The  use  of  M&S  in  manufacturing  is  aiming 
toward  a  future  “Virtual  Manufacturing” 
environment.  In  this  approach,  the  opera¬ 
tional  requirements  identified  in  the  syn¬ 
thetic  battlefield  environment  are  translated 
into  design  concepts  using  three-dimen¬ 
sional  virtual  simulations  incorporating  ge¬ 
ometry  and  performance.  These  designs  are 
passed  along  to  a  network  of  distributed 
manufacturing  simulations  which  may  reside 
throughout  a  vendor  base  (i.e.  prime  con¬ 
tractor  and  its  subcontractors)  to  identify  the 
manufacturing  processes,  facilities  and  tool¬ 
ing  requirements.  This  vendor  base  is  clos¬ 
est  lo  the  manufacturing  processes  and  is  in 
the  best  position  to  develop  cost  aiid  sched¬ 
ule  estimates.  These  estimates  may  then  be 
fed  back  up  to  provide  better  estimates  of 
costs  and  schedules  to  support  trade-offs  and 
the  system  level  alternative  evaluations  in 
the  COEA. 

The  virtual  manufacturing  initiative  is  in¬ 
tended  to  provide  the  ties  between  new 
product  design  concepts  and  the  processes 
necessary  to  manufacture  them  starting  in 
the  earliest  phases  of  development.  This  will: 

•  Provide  quick  and  improved  cost  and 


delivery  estimates; 

•  Smooth  the  transition  of  new  process 
technologies  into  production  facilities; 

•  Facilitate  lean  or  agile  manufacturing; 
and 

•  Facilitate  IPPD.“ 

5.2.5  Tesk  and  Evaluation 

The  purpose  of  a  test  and  evaluation  pro¬ 
gram  is  to  provide  information  for  risk  as¬ 
sessment  and  decision  making,  verify  attain¬ 
ment  of  technical  performance  specifica¬ 


tions  and  objectives,  and  verify  that  systems 
are  operationally  effective  and  suitable  for 
their  intended  use.  Test  planning  begins 
in  Phase  0,  CED,  resulting  in  the  initial 
TEMP  at  milestone  I.  Models  and  simula¬ 
tions  supporting  the  development  (DT)  or 
operational  test  (OT)  programs  must  be  dis¬ 
cussed  in  the  TEMP.  For  DT,  the  program 
must  “List  all  models  and  simulations  to  be 
used  and  explain  the  rationale  for  their 
use”.^’  For  OT,  the  TEMP  must  “Identify 
planned  sources  of  information  (e.g.,  devel¬ 
opment  testing,  testing  of  related  systems, 
modeling,  simulation,  etc.)  that  may  be  used 
by  the  operational  test  agency  to  supplement 
this  phase  of  operational  test  and  evalua- 


Figure  5-4.  Missle  Data  Requirements  and  Test  Assets 
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tion.  Whenever  models  and  simulations  are 
to  be  used,  explain  the  rationale  for  their 
credible  use”.‘* 

5.2.5.1  Developmental  Test  and  Evalua¬ 
tion  (DT&E) 

Weapon  systems  being  developed  today  are 
increasingly  more  complex — technology  is 
advancing,  the  ability  to  process  more  in¬ 
formation  is  rapidly  growing  and  the  per¬ 
formance  of  systems  is  increasing.  As  an  ex¬ 
ample,  consider  the  illustration  in  Figure  5- 
4’’  of  available  test  assets  and  data  require¬ 
ments  for  missile  development  programs 
over  the  last  40  years.  There  has  been  a  sig¬ 
nificant  increase  in  missile  complexity  and 
data  requirements,  but  this  increase  in  mis¬ 
sile  complexity  has  not  been  accompanied 
by  a  corresponding  increase  in  missile  launch 
assets  -  because  of  tighter  program  cost  and 
schedule  constraints. 

Figure  5-5  illustrates  this  example  further.^ 
This  figure  summarizes  the  number  of  test 
firings  for  several  Sidewinder  models  (AIM- 
9  series  missile)  developed  over  the  last  30 
years.  Most  of  the  upgrades  represent  what 


would  now  be  called  pre-planned  product 
improvements,  involving  changes  only  to  the 
missile  seeker.  (The  exception  was  the  AIM- 
9L  which  included  a  new  warhead  and  fuse 
along  with  seeker  improvements.)  The  fir¬ 
ings  shown  represent  the  total  number  of 
firings  from  research  and  development 
through  OT&E.  The  downward  trend  was 
driven  by  cost  and  schedule.  Simulations 
were  used  to  maintain  or  increase  the  level 
of  understanding  of  system  performance 
even  though  test  need  decreased. 

Simulations,  therefore,  are  used  to  “bridge 
the  gap”  between  the  ever-increasing  data 
requirements  and  the  relatively  constant,  or 
even  decreasing  available  test  assets.  Spe¬ 
cifically,  simulations  can  be  used  for: 

•  Pre-test  planning  -  Insuring  that  th^ 
tests  to  be  conducted  are,  indeed,  the  most 
critical  and  verify  instrumentation  plans. 
Simulations  can  be  used  to  identify  the  criti¬ 
cal  test  points  on  which  to  focus  the  live  tests. 
Data  from  the  simulation  can  be  used  prior 
to  actual  testing  to  check  out  and  exercise 
the  data  reduction  processes. 


Sidewinder  Firing  History 

Sidewinder  Model 

Development  Time 

Number  of  Firings 

Frame 

AIM-9D 

1960-1964 

129 

AIM-9L 

1972-1975 

69 

AiM-9M 

1978-1981 

35 

AIM-9-8/9 

1991-1993 

21 

Figure  5-5.  Sidewinder  Firing  History 
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•  Mission  rehearsal  -  “Walking  through” 
the  test  from  initial  launch  conditions  to  give 
confidence  that  tests  will  be  successful.  One 
can  use  actual  hardware  in  captive  carry 
being  stimulated  with  threat  simulators  to 
check  out  the  system  and  tactics  prior  to  test. 

•  Post-test  analysis  -  Taking  the  raw  test 
data  and  extracting  the  critical  performance 
parameters. 

•  Augment  actual  tests  -  Running  large 
numbers  of  simulations  over  many  condi¬ 
tions  for  which  test  assets  are  unavailable 
or  when  environmental,  political,  resource 
or  safety  constraints  make  testing  infeasible. 
For  example,  over  a  six  month  period  for 
one  missile,  a  total  of  4,280  HWIL  simula¬ 
tion  runs  versus  7  actual  launches  were  con¬ 
ducted.^' 

•  Risk  Reduction  -  Conducting  simula¬ 
tions  to  reduce  program  “political”  and  tech¬ 
nical  risks. 

—  Political  risk  reduction  -  Programs 
are  increasingly  under  the  scrutiny  from 
all  levels,  and  managers  can  ill  afford  the 
risk  of  a  live  test  failure.  Simulations  to 
conduct  mission  rehearsals  and  checkout 
of  the  actual  test  items  can  reduce  this 
risk. 

—  Technical  risk  reduction  -  Simula¬ 
tions  allow  developers  to  evaluate  far 
more  design  alternatives  over  more  con¬ 
ditions  in  shorter  time  periods  than  live 
tests.  This  allows  identification  and  cor¬ 
rection  of  technical  problems  early  in  a 
program;  resulting  in  a  design  that  better 
meets  technical  and  operational  require¬ 
ments.  An  example  of  this  latter  case  is 
the  use  of  HW/SWIL  simulations. 

Guidelines  on  the  use  of  M&S  in  support  of 


DT&E  may  be  found  in  the  Modeling  and 
Simulation  Master  Plan^^  prepared  by  the 
US  Army  Test  and  Evaluation  Command 
(TECOM).  This  plan  discusses  the  roles  of 
the  various  participants  in  Army  T&E  ac¬ 
tivities,  and  provides  a  vision  for  the  ad¬ 
vancement  of  M&S  to  increase  the  efficiency 
and  cost  effectiveness  of  T&E. 

5.2.5.2  Operational  Test  and  Evaluation 
(OT&E) 

An  OT&E  is  a  comprehensive  process  which 
uses  analytical  studies,  analysis,  component 
tests  and  actual  weapon  system  tests  in  a 
complimentary  manner.  In  accordance  with 
Title  10,  U.S.  Code,  “The  term  operational 
test  and  evaluation  ...  does  not  include  an 
operational  assessment  based  exclusively  on 
(a)  computer  modeling;  (b)  simulation;  or 
(c)  an  analysis  of  system  requirements,  en¬ 
gineering  proposals,  design  specifications, 
or  any  other  information  contained  in  pro¬ 
gram  documents.”” 

However,  this  does  not  mean  that  models 
and  simulations  do  not  have  a  role  in  OT&E. 
Constraints  on  testing  such  as  cost,  security, 
safety,  ability  to  portray  threats,  treaty  con¬ 
straints,  limitations  on  test  instrumentation, 
number/maturity  of  test  articles,  test  space 
and  lack  of  representative  terrain  or  weather 
may  preclude  a  comprehensive  evaluation 
based  on  field  testing  alone.  M&S  tools  can 
augment  or  complement  the  actual  field 
tests  to  provide  decision-makers  with 
needed  information  which  otherwise  would 
not  be  available. 

According  to  DOT&E  Policy,  dated  24  Jan 
1989,  appropriate  uses  of  M&S  include  test 
planning;  test  data  analysis  and  evaluation 
to  augment,  extend  or  enhance  test  results; 
tactics  development;  and  early  operational 
assessments  of  expected  capabilities.” 
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specifically,  the  policy  states:  “Ideally,  the 
user,  developer,  and  tester  would  agree  on 
the  M&S  needed  for  operationally-oriented 
assessments  for  a  system  under  consider¬ 
ation  not  later  than  Milestone  I.”  This  policy 
also  reiterates  the  importance  of  describing 
plans  in  the  TEMP  for  the  use  of  models 
and  simulations  in  OT&E  to  augment,  ex¬ 
tend  or  enhance  field  test  results. 

Credibility  is  a  key  part  of  successful  use  of 
the  M&S  in  supporting  OT&E.  This  in¬ 
cludes  an  acceptable  M&S  approach;  confi¬ 
dence  in  the  models,  users,  methodology, 
and  results;  and  a  robust  VV&A  process. 
Appendix  B  of  the  DTO&E  policy  provides 
a  list  of  issues  that  should  be  addressed  to 
provide  evidence  of  credible  models  and 
simulations  in  OT&E.  The  reader  is  also 
referred  to  further  discussion  of  credibility 
contained  in  Chapter  6  of  this  guidebook. 

The  Service’s  operational  test  agency  is  ac¬ 
countable  for  the  OT  results  that  they  re¬ 
port  and,  hence,  results  of  any  M&S  it  uses 
in  support  of  OT&E.  As  an  example,  the 
Army’s  guidance  on  M&S  to  support  OT 
identifies  the  Commander  of  Army  Opera¬ 
tional  Test  and  Evaluation  Command 
(OPTEC)  as  the  accrediting  official  for 
models  and  simulations  used  within  that 
organization.”  Although  no  formal  docu¬ 
mentation  was  found,  discussions  with  the 
Air  Force  and  Navy  OT&E  agencies  imply 
that  the  same  policy  is  followed  within  the 
other  Services. 

In  the  future,  the  test  community,  using  ad¬ 
vanced  distributed  simulation  (ADS),  will 
be  able  to  conduct  live  tests  which  are  net¬ 
worked  to  geographically  dispersed  human- 
in-the-loop  simulations  within  a  synthetic 
environment.  This  provides  for  a  realistic 
test/simulation  in  a  war-like  environment 
with  a  variety  of  friendly  and  hostile  com¬ 


batants.  Currently,  in  the  OT&E  environ¬ 
ment,  distributed  simulation  is  more  useful 
in  test  planning  than  actual  conduct  of  tests 
because  of  issues  such  as  VV&A  of  entities 
within  a  distributed  environment. 

A  Joint  Advanced  Distributed  Simulation 
Joint  Feasibility  Study  (JADS/JFS)  is  cur¬ 
rently  underway  to  address  issues,  prin¬ 
ciples,  procedures  and  practices  for  the  in¬ 
creased  use  of  distributed  simulation  sup¬ 
port  to  both  developmental  and  operational 
tests  and  evaluations.” 

5.2.5  J  Live  Fire  Testing  (LFT) 

Title  10  of  the  US  Code^^  requires  realistic 
survivability  testing  of  covered  systems  (or 
product  improvement  programs)  and  lethal¬ 
ity  testing  for  major  munitions  programs 
prior  to  proceeding  beyond  low  rate  initial 
production.  Examples  of  M&S  supporting 
LFT  include:  aircraft  and  missile  flight  path 
generation;  detection,  tracking,  and  shoot¬ 
ing  performance  of  artillery;  warhead-tar¬ 
get  fragment  interactions;  penetration  me¬ 
chanics  and  failure  mode  analysis. 

Evaluations  of  materials,  fuel  system  design, 
internal  routing  of  lines  and  cables,  etc.  are 
accomplished  using  models  and  simulations 
which  can  facilitate  “design  for  survivabil¬ 
ity”  early  in  development  before  hardware 
is  produced  and  tested.  The  Survivability/ 
Vulnerability  Information  Analysis  Center 
(SURVIAC)”  is  a  centralized  information 
resource  for  information  on  survivability  and 
lethality.  The  SURVIAC  has  an  inventory 
of  models  and  simulations  and  can  provide 
programs  with  technical  advice. 

The  acquisition  manager  should  recognize 
that  the  use  of  M&S  complements  the  T&E 
activities.  It  has  been  recommended  that  an 
integrated  model-test-model  approach  be 
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implemented  in  development  programs  with 
three  aims  in  mind: 

•  Ensure  models  and  simulations  still 
meet  the  developer’s  needs; 

•  Use  models  and  simulations  to  iden¬ 
tify  critical  tests,  data  requirements,  analyze 
data  and  reduce  the  amount  of  actual  test¬ 
ing;  and 

•  Ensure  every  test  serves  the  dual  pur¬ 
pose  of  evaluating  system  performance  and 
validating  the  models  and  simulations.^’ 


Such  an  approach  has  been  common  in  elec¬ 
tronic  combat  system  development  pro¬ 
grams.  These  development  programs  em¬ 
ploy  heavy  use  of  models  and  simulations 
prior  to  testing  within  integration  laborato¬ 
ries,  simulation  facilities  and  finally,  in  the 
op«n-air.  Testing  is  then  followed  by  further 
modeling  to  analyze  test  data  and  extract 
the  MOP  and  MOE.'’ 

This  concept  of  model-test-model  is  appli¬ 
cable  to  all  system  development  programs 
and  an  adaptation  of  the  above  two  philoso¬ 
phies  is  illustrated  in  Figure  5-6. 


Model-Test-Model  Approach  to  Development 


SYSTEM 

DEVELOPMENT 


MODELING  &  SIMULATION 
(PRE-TEST) 

•  ID  CRmCAL  TESTS 

•  DEVELOP  TEST  SCENARIO 

•  DEFINE  INSTRUMENTATION 

•  PLAN  DATA  ANALYSIS 


•  REQUIREMENTS 

VERIFICATION/UPDATES 


•  PREDICT  TEST  RESULTS 

•  MISSION  REHEARSALS 


•  DESIGN  UPDATES 


•  HARDWARE/SOFTWARE 
CHECKOUTS  (HW/SWIL) 

I 

MODELING  &  SIMULATION 
(POST-TEST) 


•  ANALYZE  DATA 
-COMPARE  WITH 
PRE-TEST  PREDICTIONS 


TEST 

SIMULATION) 


•  INSTRUMENT  SYSTEM 

•  CONDUCT  LIVE  TESTS 

•  COLLECTS  PROCESS  DATA 
(FOR  BOTH  TEST  RESULTS 
AND  M&S  VALIDATION) 


•  EXTRACT  AND  EXTRAPOLATE 
MOPS  &  MOEs 

•  VAUDATEAJPDATE  M&S 


Figure  5-6.  Model-Test-Mo(jel  Approach  to  Development 
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5.2.6  Logistics  Support 

Models  and  simulations  support  logistics 
analyses  across  the  system  life  cycle — from 
defining  system  level  supportability  concepts 
to  reliability,  availability  and  maintainabil¬ 
ity  design  requirements — and  eventually 
modeling  actual  operational  capability  dur¬ 
ing  operations  and  support. 

An  overview  of  the  way  in  which  M&S  can 
be  used  to  support  logistics  follows.'*' 

Early  activities  in  the  logistics  community 
include  building  the  baseline  comparison 
system  which  can  be  used  along  with  M&S 
to  do  a  comparative  analysis  for  the  pro¬ 
posed  new  system;  identify  supportability, 
cost,  and  readiness  drivers;  and  estimate  the 
operations  and  support  portion  of  the  life 
cycle  costs. 

In  DemVal,  as  the  weapon  system  becomes 
more  defined  at  the  subsystem  level,  level 
of  repair  analysis  (LORA)  models  are  used 
to  identify  candidate  areas  for  interface  be¬ 
tween  logistics  and  design.  These  analyses 
help  define  trade-offs  in  manpower;  reliabil¬ 
ity,  availability,  and  maintainability;  and  al¬ 
ternate  maintenance  concepts  and  their  ef¬ 
fects  on  supportability  for  specific  subassem¬ 
blies.  Using  these  models  to  quantify  the 
impacts  on  support,  the  logisticians  can  in¬ 
terface  with  the  designers  to  produce  designs 
that  lead  to  reduced  overall  support  costs. 
The  LORA  models  will  then  be  used  for  the 
actual  repair  level  decision-making  and  form 
the  basis  for  the  system  maintenance  plan. 

In  EMD,  models  will  be  used  to  analyze  re¬ 
pair  tasks  and  identify  the  requirements  in 
the  ILS  elements  for  each  component.  The 
results  of  these  analyses  form  a  data  reposi¬ 
tory,  the  Logistics  Support  Analysis  Records 
(LSARs),  which  can  be  used  in  the  detailed 


identification  of  logistics  resource  require¬ 
ments  for  each  element  of  logistics,  as  well 
as  projected  readiness  modeling.  Among  the 
models  used  are  provisioning  models  to  de¬ 
termine  initial  spares  requirements  and  the 
optimum  spare  parts  and  quantities  neces¬ 
sary  to  attain  end  item  operational  availabil¬ 
ity  at  the  least  cost. 

Early  in  development,  engineering  estimates 
of  component  failure  rates  are  used  in  the 
models.  As  the  system  matures  and  is  even¬ 
tually  fielded,  test  data  and  actual  opera¬ 
tional  data  become  available.  This  data  re¬ 
places  the  initial  estimates  on  failure  and 
repair  in  the  LSARs.  During  O&S,  this  in¬ 
formation  can  be  used  in  models  and  simu¬ 
lations  to  evaluate  actual  system  readiness, 
adjust  provisioning  levels  or  support  system 
operational  planning.  Models  and  simula¬ 
tions  also  find  use  in  this  phase  to  evaluate 
the  supportability  impacts  of  proposed  ECPs 
or  modifications  to  the  system. 

The  ILS  elements  and  logistics  support 
analysis  (LSA)  tasks  are  supported  by  an 
assortment  of  models  or  simulations.  One 
source  of  information  on  these  models  and 
simulations  is  the  “Logistic  Support  Analy¬ 
sis  Techniques  Guide”."*^  This  guide  is  pre¬ 
pared  by  U.S.  Army  Materiel  Command 
Logistics  Support  Agency  (LOGSA),  which 
is  designated  as  the  DoD  lead  agency  for 
LSA.**^  This  guide  contains  descriptions  of 
105  models  or  simulations  cross  referenced 
to  LSA  tasks  and  ILS  elements,  along  with 
points  of  contact  for  each  model. 

Another  source  of  information  is  the  Sup¬ 
portability  Investment  Decision  Information 
Analysis  Center  (SIDAC)  which  maintains 
a  small  number  of  logistics  models  and  can 
provide  assistance  in  preparing  and  running 
those  models  and  using  assorted  logistics 
related  data  bases."*^ 
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5.2.7  IVaining 

Training  is  integral  to  achieving  and  main¬ 
taining  force  readiness.  Despite  reductions 
in  force  structure  and  annual  operating 
funds,  the  services  are  determined  to  main¬ 
tain  their  “warfighting  edge”  with  superior 
training.  Throughout  DoD,  simulation  in 
support  of  training  spans  all  of  the  classes 
of  simulation. 

Wargaming  is  used  to  train  battle  staff  in 
planning  and  execution  of  tactics  from  indi¬ 
vidual  system  level  through  combined  assets 
applications.  This  is  often  accomplished  us¬ 
ing  constructive  models  representing  sys¬ 
tems  or  groups  of  systems  or  may  even  be 
linked  to  live  systems.  Facilities  which  sup¬ 
port  such  simulations  may  allow  multiple 
participants  to  interact  and  provide  record¬ 
ing  of  events  for  subsequent  data  analysis 
and  debriefing  of  participants. 

Virtual  simulators  such  as  weapon  system 
simulators  (aircraft,  tank,  ship,  etc.)  are 
commonly  used  for  training.  These  simula¬ 
tors  immerse  operators  in  a  realistic  envi¬ 
ronment  (visual,  aural,  motion)  allowing 
them  to  perform  a  mission  as  if  they  were  in 
the  actual  vehicle,  thereby  receiving  com¬ 
bat  realistic  training.  Another  example  of 
immersing  the  operator  in  a  virtual  environ¬ 
ment  might  be  an  air  defense  simulator 
which  allows  operators  at  multiple  consoles 
to  track,  identify,  allocate  and  control  weap¬ 
ons  using  command  and  control  formats 
obtained  from  other  simulated  platforms. 
Weapon  characteristics  might  be  provided 
via  computer  generated  weapon  simulations. 

Live  simulations  in  support  of  training  in¬ 
clude  the  Army  National  Training  Center  at 
Ft.  Irwin,  the  Navy  “Strike  University”  at 
Fallon  Naval  Air  Station,  the  Air  Force  “Red 
Flag”  at  Nellis  AFB,  and  the  Marine  Corps 


Air-Ground  Combat  Center  at  Twentynine 
Palms.  These  simulations  allow  participants 
to  operate  systems  under  environmental 
conditions  which  approach  real  life  in  com¬ 
bat.  Data  gathered  during  instrumented  ex¬ 
ercises  can  be  used  to  debrief  participants, 
and  can  provide  the  system  acquisition  com¬ 
munity  valuable  information  on  perfor¬ 
mance  of  weapon  systems  and  human  inter¬ 
action  during  close  to  real  combat  condi¬ 
tions. 

The  future  application  of  simulation  to  train¬ 
ing  will  involve  a  combination  of  live  and 
virtual  participants  within  synthetic  environ¬ 
ments  and  will  allow  for  training  with  indi¬ 
vidual  participants  geographically  distrib¬ 
uted.  This  will  become  a  reality  in  the  train¬ 
ing  community  when  the  Navy/Air  Force 
Joint  Tactical  Combat  Training  System 
(JTCTS),  the  Navy  Battle  Force  Tactical 
Training  (BFTT)  System,  and  the  Army’s 
Close  Combat  Tactical  Trainer  (CCTT)  are 
fielded. 

The  JTCTS  is  a  training  instrumentation 
system  that  will  revolutionize  the  way  the 
Navy  and  Air  Force  conduct  air  to  air,  air  to 
ground,  ship  air/surface/submarine  warfare 
and  joint  training.  It  is  a  live  training  simu¬ 
lation  that  combines  simulated  and  real  tar¬ 
gets  detected  and  displayed  by  platform  sen¬ 
sors.  A  core  computer  system  that  performs 
scenario  development,  scenario  transmis¬ 
sion,  data  logging  and  post-exercise  debrief. 
Each  participant  (aircraft,  ships,  subma¬ 
rines)  will  have  an  instrumentation  package 
and  data  links  to  inject  scenario  events  into 
the  participant’s  combat  systems  through 
simulation  or  stimulation. 

The  Navy  BFTT  is  a  virtual  training  system 
that  uses  the  actual  ship’s  combat/sensor 
system  as  the  training  system.  The  BFTT 
computer  on  each  ship  will  stimulate  the 
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radar,  sonar,  electronic  warfare  equipment 
and  communications  suite  with  scenario 
events  controlled  by  the  BFTT  computer  for 
single  ship  training;  from  another  ship  for 
multi-ship  training;  or  from  a  shore  based 
BFTT  computer  for  fleet  or  joint  virtual 
training  on  the  synthetic  battlefield.  This 
system  also  provides  a  debrief  for  partici¬ 
pants  to  gain  maximum  benefit  from  each 
training  scenario. 

The  Army  CCTT  is  a  network  of  manned 
simulators  providing  combined  arms  and 
collective  training  using  force-on-force  free- 
play  simulation  on  an  electronic  battlefield. 
The  manned  simulators  will  include  the  M-1 
series  tank,  M-2/M-3  Bradley  fighting  ve¬ 
hicles,  FIST-V  vehicles  and  dismounted  in¬ 
fantry.  Fbced  and  mobile  CCTT  systems  are 
planned.  The  JTCTS,  BFTT,  and  CCTT  will 
rely  on  the  distributed  interactive  simula¬ 
tion  (DIS)  communications  standards  for 
data  communications,  which  are  discussed 
in  more  detail  in  Chapter  6. 

The  PM  should  aim  to  maximize  the  use  of 
simulations  between  weapon  system  and 
training  system  development.  One  example 
of  simulators  being  developed  as  weapon 
system  trainers  serving  an  additional  func¬ 
tion  for  the  acquisition  program  is  in  the  B-2 
aircraft  program.  As  part  of  the  Operational 
Flight  Program  (OFP)  software  develop¬ 
ment  process,  the  B-2  aircraft  program  used 
the  weapon  system  trainer  as  a  systems  in¬ 
tegration  lab  to  compile  and  check  run  the 
software  in  conjunction  with  other  real  and 
synthetic  data.  After  any  debugging,  the 
OFP  was  returned  to  the  Flight  Test  Center 
to  be  certified  for  flight. 

The  above  discussion  provides  the  acquisi¬ 
tion  manager  insight  into  both  the  present 


and  planned  applications  of  models  and 
simulation  to  training.  In  many  cases,  the 
models  and  simulations  which  support  the 
development  of  the  weapon  system  can  be 
used  to  support  the  training  systems,  be  they 
system  simulators,  or  distributed  training 
systems  combining  live,  virtual  and  construc¬ 
tive  simulations.  Currently,  models  and 
simulations  for  training  purposes  are  often 
developed  separately  by  another  software 
development  activity.  The  PM  should  not 
have  to  pay  for  these  simulations  twice — an 
integrated  M&S  plan  during  the  CED  phase 
can  help  the  transition  of  simulations  be¬ 
tween  the  system  and  its  training  simulator. 

53  Summary 

This  Chapter  provided  an  overview  of  the 
use  of  models  and  simulations  across  the 
acquisition  life  cycle  and  in  specific  acquisi¬ 
tion  activities.  The  challenge  for  PMs  in  us¬ 
ing  these  models  and  simulations  efficiently 
is  to: 

•  Integrate  the  use  of  M&S  within  pro¬ 
gram  planning  activities  and  documentation; 

•  Plan  for  life  cycle  application,  support 
and  reuse  of  models  and  simulations;  and 

•  Integrate  M&S  across  the  functional 
disciplines. 

To  aid  PMs  in  their  planning  for  the  use  of 
models  and  simulations,  Appendbc  G  con¬ 
tains  templates  covering  some  of  the  acqui¬ 
sition  activities  discussed  in  this  chapter. 
These  templates  should  be  considered  as 
guidelines  only;  each  program  must  tailor 
the  models  and  simulations  it  uses  to  the 
specific  activities  to  be  accomplished  within 
that  program. 
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MODELING  AND  SIMULATION 

ISSUES 


6.1  Introduction 

Over  the  years,  a  migration  has  occurred 
from  live  toward  constructive  and  virtual 
models  and  simulations.  With  this  migration, 
software  has  become  an  ever-increasing  part 
of  models  and  simulations.  This  chapter  dis¬ 
cusses  certain  technical  aspects  of  software 
development,  use  and  management  which 
play  key  roles  in  planning  for  the  use  of  these 
models  and  simulations. 

0.2  Credibility  of  Models  and  Simulations 
(or  “How  can  I  believe  what  I’m  seeing?”) 

The  last  thing  a  manager  wants  to  do  is  to 
take  a  recommendation  up  to  a  decision¬ 
maker,  be  asked,  “Why  should  I  believe  this 
analysis?”,  and  not  be  able  to  substantiate 
the  credibility  of  the  underlying  information. 
Not  only  would  the  manager  have  wasted 
valuable  time  and  money,  but  the  credibility 
of  future  program  actions  may  have  been 
jeopardized. 

By  its  very  nature,  a  model  is  an  abstraction 
of  the  real  world.  Factors  are  often  not  in¬ 
cluded  in  a  model;  based  upon  an  assessment 
of  the  relevance  of  specific  factors  to  the 


performance  of  the  model.  The  perceived 
relevance  of  a  given  factor  is  also  balanced 
against  the  cost  and  complexity  of  including 
the  factor  and  modeling  its  effects.  As  a  re¬ 
sult.  modelers,  analysts  and  users  routinely 
make  assumptions  about  the  relevance  of 
specific  factors  to  the  performance  of  the 
model.  This  assessment  looks  at  the  value 
that  would  ;)e  added  by  their  inclusion,  and 
the  complexity  of  effort  involved  in  includ¬ 
ing  them.  For  example,  in  most  simulations, 
certain  human  sensory  perceptions,  such  as 
smell,  heat  and  cold  are  omitted,  as  being 
either  analytically  irrelevant  or  not  cost-ef¬ 
fective  to  model. 

Naturally,  these  assessments  are  effected  by 
how  well  the  factor  and  its  impact  on  the 
environment  being  modeled  are  understood. 
Early  representations  of  a  system  are  based 
upon  what  is  known,  at  that  time,  of  the  tech¬ 
nology  and  related  phenomenologies.  Mod¬ 
els  evolve  in  a  spiral  fashion  as  knowledge 
of  these  aspects  is  gained. 

So,  how  do  we  know  that  we  haven’t  omit¬ 
ted,  or  overlooked,  something  important? 
More  importantly,  since  results  of  models 
and  simulations  are  analyzed  to  support  de- 
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cision-making,  how  do  we  convince  the  de¬ 
cision-maker  that  the  analyses  are  credible? 

Both  these  concerns  are  addressed,  in  part, 
by  ensuring  that  the  modeling  and  simula¬ 
tion  (M&S)  supporting  any  analysis  are  both 
complete  and  correct.  To  establish  this  fact, 
it  is  imperative  that  the  M&S  be  subjected 
to  scrutiny. 

A  M&S  must  be  verified,  validated,  and  ac¬ 
credited  (VV&A).  This  is  a  continuous  pro¬ 
cess  forming  an  integral  part  of  an  M&S’ 
development,  use  and  maintenance.  Cred¬ 
ible  documentation  is  also  a  critical  ingredi¬ 
ent  in  success  of  the  VV&A  process,  as  is 
the  control  of  the  overall  development  pro¬ 
cess. 

63  Verification,  Validation  and 
Accreditation  (W&A) 


63.1  What  is  W&A? 

As  Figure  6-1  depicts,  VV&A  of  a  model  or 
simulation  collectively  contribute  to  its  cred¬ 
ibility.  As  the  figure  shows,  VV&A  involves 
the  M&S  developer,  the  functional  expert 
and  the  user.  Although  the  definitions  are 
included  in  the  glossary,  this  is  probably  a 
good  place  for  a  more  focused  definition  of 
each  term. 

63.1.1  Verifleation  -  The  process  of  deter¬ 
mining  that  a  model  implementation  accu¬ 
rately  represents  the  developer’s  conceptual 
description  and  specifications. ' 

Proper  conduct  of  verification  answers  the 
question:  “Is  it  what  I  intended?” 

Verification  is  applied  at  each  stage  of  the 
life  cycle  to  ensure  that  the  products  of  that 
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Figure  6-1 .  Verification,  Validation  and  Accreditation 
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stage  accurately  implement  the  output  from 
the  previous  stage  and  contribute  to  the  over¬ 
all  goal.  It  requires  identification  and  exami¬ 
nation  of  explicit  and  implicit  assumptions 
and  logic  flows,  and  includes  appropriate 
data  certification. 

6.3.1.2  Validation  -  The  process  of  determin¬ 
ing  (a)  the  manner  and  degree  to  which  a 
model  is  an  accurate  representation  of  the  real 
world  from  the  perspective  of  the  intended  use 
of  the  model,  and  (b)  the  confidence  that 
should  be  placed  on  this  assessment.  ^ 

Proper  conduct  of  validation  answers  the 
question;  “How  well  does  the  model  repre¬ 
sent  what  it  claims  to  represent?” 

Validation,  too,  must  encompass  the  entire 
system,  the  model,  data  and  even  the  opera¬ 
tors  and  analysts  who  will  use  the  model  or 
its  results.  It  is  a  rigorous  process,  involving 
both  the  structure  of  a  model  or  simulation; 
as  well  as  its  output.  Validation  addresses 
both  the  fidelity  and  resolution  of  a  model 
or  simulation.  It  ensures  that  all  aspects  of 
the  real  world  which  should  be  represented 
are  accounted  for  in  sufficient  detail  to  ad¬ 
equately  establish  cause-and-effect  relation¬ 
ships.  Validation  provides  the  foundation 
for  the  accreditation  process  to  build  upon. 

6.3.1.3  Accreditation  -  The  official  certifica¬ 
tion  that  a  model  or  simulation  is  acceptable 
for  use  for  a  specific  purpose.  ^ 

Proper  conduct  of  the  accreditation  process 
answers  the  question:  “Should  I  endorse  this 
model?” 

The  agency  using  the  results  from  a  specific 
application  of  a  M&S  (the  application  spon¬ 
sor)  must  establish  a  set  of  standards,  or  ac¬ 
ceptability  criteria,  that  a  particular  M&S 
must  meet  to  be  accredited  for  a  given  use. 


It  is  important  to  recognize  that  any  use  of 
the  results  of  a  M&S  is  considered  de  facto 
accreditation  by  the  M&S  application  spon¬ 
sor.  Obviously,  the  preferred  method  of  ac¬ 
creditation  involves  a  determination,  before 
use,  that  the  M&S  is  appropriate  for  that  use. 

Verification  and  validation  ( V&V)  consist  of 
varying  levels  and  types  of  technical  evalua¬ 
tions  for  the  model,  its  development  process, 
as  well  as  the  data  used  to  run  it. 

Accreditation  is  a  management  determina¬ 
tion,  and  is  largely  based  upon  the  technical 
evidence  and  audit  trail  resulting  from  the 
V&V  process. 

6  J.2  Why  is  W&A  necessary? 

The  purpose  of  VV&A  is  to  establish  the 
credibility  of  a  model  or  simulation  as  a 
source  of  data  for  analysis. 

The  output  data  from  models  are  used  to 
feed  analyses  supporting  decision-making  in 
the  DoD  acquisition  process.  These  analy¬ 
ses  define  weapon  systems  requirements, 
influence  procurement  decisions,  aid  in  en¬ 
gineering  design,  plan  and  conduct  test  and 
evaluation;  and  establish  maintenance  lev¬ 
els,  sparing,  production  rates,  etc.  Further¬ 
more,  since  output  data  from  one  model  are 
often  used  as  input  for  another,  errors  be¬ 
come  compounded  and  their  sources  un- 
traceable. 

A  1993  DoD  Inspector  General  (IG)  Audit 
found  that  as  many  as  95  percent  of  models 
and  simulations  surveyed  (and  in  use)  had 
not  fully  incorporated  VV&A.'*  The  IG’s 
report  recommended  development  of  DoD 
policy,  guidance,  standards  and  criteria  for 
VV&A  of  models  and  simulations. 

At  the  time  of  this  writing,  official  DoD 
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policy  on  VV&A  was  being  developed,  as 
were  official  Navy,  Marine  and  Air  Force 
policies.  Department  of  The  Army  Pamphlet 
(DA  PAM)  5-11,  W&A  of  Army  Models  and 
Simulations,  provides  methods,  techniques 
and  procedures  to  conduct  VV&A  for  M&S. 
However,  overall  Army  policy  with  respect 
to  development,  use  and  management  (in¬ 
cluding  VV&A)  of  M&S  is  set  forth  in  Army 
Regulation  (AR)  5-11,  Army  Model  and 
Simulation  Management  Program. 

Some  useful  information  on  VV&A  is  avail¬ 
able  from  the  Susceptibility  Model  Assess¬ 
ment  and  Range  Test  (SMART)  project 
sponsored  by  the  Joint  Technical  Coordinat¬ 
ing  Group  on  Aircraft  Survivability  (JTCG/ 
AS).  Although  focused  on  survivability  and 
vulnerability  M&S,  the  SMART  project  has 
developed  methodologies  for  VV&A  and 
conducted  research  on  VV&A  procedures 
throughout  DoD  that  may  assist  program 
offices.^ 

6.3.3  IVeatment  of  Legacy  Models 

Some  models  for  years  have  been,  and  still 
are,  workhorses  within  the  various  functional 
communities.  These  models  have  usually 
provided  information  upon  which  solid  de¬ 
cisions  have  been  based— decisions  that  have 
later  been  justified  by  actual  weapon  systems 
performance  in  testing  or  in  the  field.  It 
would  be  senseless  to  preclude  the  use  of 
these  models.  Special  provisions  within  most 
policies  must  permit  grandfathering  of  mod¬ 
els  developed  prior  to  the  implementation 
date  of  formal  VV&A  policies. 

However,  even  for  a  legacy  model,  it  is  ab¬ 
solutely  essential  that  the  accreditation  au¬ 
thority,  as  a  minimum,  step  through  the  logi¬ 
cal  thought  process  that  establishes  whether 
or  not  it  is  appropriate  for  use  in  a  certain 
application. 


6.3.4  W&A:  When  should  it  be  done? 


It  is  far  better  to  plan  and  provide  resources 
for  W&A  up-front,  than  to  attempt  recov¬ 
ery  later. 


The  VV&A  of  models  is  neither  an  after¬ 
thought,  nor  a  one-shot  affair.  An  integral 
part  of  the  model’s  development  process 
must  be  VV&A.  One  benefit  of  this  ap¬ 
proach  is  that  it  becomes  tougher  to  yield  to 
the  temptation  of  cutting  out  VV&A,  while 
continuing  apace  with  model  development. 
Otherwise,  VV&A  is  in  danger  of  being 
viewed  as  an  unnecessary  evil,  and  being  cut 
when  resources  decline. 

As  a  process,  it  is  also  much  simpler  to  con¬ 
duct  V&V  as  a  model  is  being  developed  or 
modified,  than  to  try  to  re-construct  the  in¬ 
formation  required  later. 

Managers  who  prefer  not  to  live  on  the  edge, 
should  ensure  the  V&V  status  of  the  mod¬ 
els  and  simulations  used  to  feed  the  analy¬ 
ses  supporting  their  decisions.  The  M&S 
must  also  be  periodically  re-accredited  ( AR 
5-11  specifies  every  five  years),  to  ensure  that 
changes  in  the  world  around  it  have  not  im¬ 
pacted  its  credibility.  Re-accreditation  is  not 
only  when  a  model  is  changed  or  used  for  a 
new  application.  This  also  gives  the  model 
manager  an  opportunity  to  review  recent 
usage  of  the  model,  and  make  a  determina¬ 
tion  concerning  its  continued  utility  to  the 
users.  The  requirement  for  periodic  re-ac¬ 
creditation  also  enforces  the  concept  of 
V&V  as  continuous  processes  through  the 
life  cycle  of  an  M&S. 

Acquisition  managers  must  understand  that 
the  emphasis  on  VV&A  is  increasing.  Data 
used  in  decision-making  must  be  defensible. 
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M&S  Confidence  Assessment  Scope-of-Evidence 


Figure  6-2.  M&S  Confidence  Assessment  Scope-of-Evidence 


It  is  a  lot  easier  to  challenge  the  credibility 
of  analyses  on  the  basis  of  the  VV&A  status 
of  underlying  M&S;  rather  than  to  challenge 
the  analyses  themselves. 

6.3.5  W&A:  Getting  it  done. 

The  thrust  of  VV&A  is  to  establish  a  degree 
of  confidence  in  the  analyses  resulting  from 
the  use  of  an  M&S.  The  scope  of  technical 
evidence  that  must  be  presented,  before  ex¬ 
pecting  a  management  decision  with  respect 
to  the  credibility  of  an  M&S  as  a  tool,  is  de¬ 
picted  in  Figure  6-2*'. 

Acquisition  managers  who  require  devel¬ 
opment  of  M&S  will  be  well-advised  to 
ensure  that  the  following  four  steps  are 


part  of  the  process. 

1.  Contracts  call  out  the  appropriate 
V&V  tasks  and  criteria. 

2.  Resources  to  accomplish  the  required 
V&V  activities  (in-house  and  contractually) 
are  provided. 

3.  Data  are  collected  at  appropriate 
points  for  validation,  and  that  the  resources 
required  for  test  activities  are  provided. 

4.  Program  schedules  allow  for  the  per¬ 
formance  of  V&V  tasks  required,  including 
the  administrative  processing  requirements 
to  obtain  formal  sign-off  by  the  accredita¬ 
tion  authority. 
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6.3.6  Who’s  Responsible  for  W&A? 

Sponsor,  developer,  requester  or  user. 
Whether  the  developer  or  the  sponsor,  par¬ 
ticipation  in  the  process  is  critical.  Every¬ 
one  has  a  vested  interest  in  ensuring  that 
credible  and  supportable  decisions  result 
from  the  analyses  fed  by  the  use  of  a  model. 

Model  developers  are  most  familiar  with  the 
model’s  construction,  and  are  normally  best 
equipped  to  verify  that  it  has  been  built  the 
way  it  was  intended.  This  does  not  negate 
the  requirement  for  independent  verifica¬ 
tion  of  the  design. 

Likewise,  functional  experts  are  normally 
the  best  equipped  to  determine  whether  the 
model  portrays  the  real  world  accurately 
enough.  However,  here  too,  an  independent 
validation  is  required. 

Accreditation  is  basically  a  matter  of  accep¬ 
tance.  It  answers  the  question,  “Am  I  going 
to  accept  the  output  of  the  model?”  This  is 
the  model  user’s  decision,  since  it  is  normally 
this  person  who  has  to  defend  any  recom¬ 
mendation  resulting  from  analyses  of  the 
model’s  output.  It  is  for  this  reason  that  the 
model  sponsor/requester/user  remains  re¬ 
sponsible  for  ensuring  that  V&V  has  been 
performed  on  the  model  or  simulation.  Cer¬ 
tainly,  one  way  to  accomplish  this  is  to  in¬ 
clude  a  requirement  for  documentation  of 
V&V  activities  and  stipulate  acceptability 
criteria  in  the  Request  For  Proposal  (RFP) 
and  Statement  of  Work  (SOW). 

When  designing  the  VV&A  plan  to  gather 
relevant  information  to  support  the  accredi¬ 
tation,  the  question  of  “How  the  model(s) 
will  integrate”  must  be  considered. 


6.4  Integration 

This  guide’s  focus,  with  regard  to  integration, 
principally  refers  to  the  extent  to  which  mod¬ 
els  and  simulations  are  able  to  cross  the  barri¬ 
ers  between  the  various  functional  communi¬ 
ties.  To  effectively  support  weapon  systems 
acquisition,  models  must  be  able  to  support 
the  implementation  of  Integrated  Product  and 
Process  Development  (IPPD).  Their  principal 
contribution  is  in  improving  the  cross-func¬ 
tional  communications  among  members  of 
Integrated  Product  Teams  (IPT). 

Existing  models  and  simulations  used  in  sup 
port  of  acquisition  are  mostly  of  the  type  that 
support  analyses  within  the  individual  ma¬ 
jor  functional  areas;  concepts  and  require¬ 
ments  development;  design  and  engineer¬ 
ing;  logistics,  manufacturing  and  production; 
test  and  evaluation;  training;  and  nrogram 
management. 

There  are  distinct  cases  of  limited  o:  !.>s.s-fune- 
tionality  between  selected  areas.  However, 
for  the  most  part,  the  use  of  models  and 
simulations  appear  to  be  functionally 
stovepiped.  An  acquisition  environment 
supporting  increased  integration  among 
functional  areas  is  needed  to  break  out  of 
the  stovepipes.  This  is  an  envisioned  infra¬ 
structure  that  would  support  the  informa¬ 
tion  needs  of  users,  developers  and  decision¬ 
makers  to  perform  integrated  cross-func¬ 
tional  analyses  throughout  the  acquisition 
process. 

As  an  objective,  it  is  not  enough  for  IPT  mem¬ 
bers  to  be  able  to  just  accept  the  validity  of 
output  from  models  in  other  functional  disci¬ 
plines  —  they  must  be  able  to  use  the  output 
to  asse.ss  the  cross-functional  implications. 

The  Director  of  Defense  Research  and 
Engineering’s  (DDR&E)  Acquisition  Task 
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Force  on  Modeling  and  Simulation 
(ATFM&S)  formed  an  IPT  Focus  Group  to: 

•  Identify  needs  for  models  and  simula¬ 
tions  and  related  tools  that  support  IPTs  in 
an  integrated  acquisition  environment;  and 

•  Develop  technology  and  non-technology 
recommendations  that  meet  those  needs. 

The  final  report  of  the  ATFM&S  expresses 
the  need  to  develop  links  between  various 
types  of  design  and  engineering  models  and 
simulations.  Some  of  the  other  key  acquisi¬ 
tion  functional  areas  need  to: 

•  Evaluate  performance  of  alternative 
designs  in  combat  environments; 


•  Assess  logistics  impact  of  alternative 
designs; 

•  Evaluate  manufacturing  and  pro- 
ducibility  implications  of  alternative  designs; 
and 

•  Evaluate  cost,  schedule  and  perfor¬ 
mance  impacts  of  alternative  designs. 

The  creation  of  such  linkages  permits  the 
evaluation  of  alternative  weapon  system  de¬ 
signs  on  the  basis  of  the  implications  across 
other  critical  functional  areas — in  effect  pro¬ 
viding  the  means  for  moving  toward  a  broader 
concurrent  engineering  framework.  Figure 
6-3  depicts  the  concept  of  increasing  cross¬ 
functional  integration  of  acquisition  M&S. 


Increasing  Cross-Functional  Integration 
of  M&S  in  Systems  Acquisition 
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Figure  6-3.  Increasing  Cross-Functional  Integration  of  M&S  in  Systems  Acquisition 
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With  increasing  cross-functionality,  comes 
the  need  for  an  increasing  commitment  to  a 
shared  vision.  Besides  the  organizational 
structure,  such  a  shared  vision  would  also 
need  a  commitment  to  the  use  of  standards, 
when  appropriate.  Models  supporting  the 
various  acquisition  functions  would  need  to 
be  able  to  interact  with  each  other,  sharing 
data  and  information  which  will,  in  effect, 
allow  them  a  common  view  of  the  same  elec¬ 
tronic  space. 

6.5  Advanced  Distributed  Simulation  (ADS) 

A  significant  emerging  capability  of  real-time 
simulation  is  the  ability  to  synthetically  cre¬ 
ate  large  environments  within  which  large 
numbers  of  subjects  can  interact.  These  vir¬ 
tual  worlds  are  made  possible  by  electroni¬ 
cally  linking  individual  simulations  such  that 
they  cooperate  in  a  shared  representation 
of  space.  Termed  ADS,  the  movement  to  cre¬ 
ate  these  virtual  worlds  is  being  driven  by 
DoD’s  attempts  to  harness  the  potential  op¬ 
portunities  resulting  from  the  explosive  growth 
in  information  processing  technology. 

The  capability  demonstrated  by  ADS  is  the 
type  required  to  truly  integrate  the  various 
functional  aspects  of  weapon  system  acqui¬ 
sition.  The  Defense  Science  Board  has  stud¬ 
ied  the  potential  impact  of  ADS  on 
prototyping,  and  determined  that  this  tech¬ 
nology  can  provide  the  means  to  “transform 
the  acquisition  system  from  within.”^ 

The  concept  of  ADS  carries  with  it  two  dis¬ 
tinct  characteristics.  The  first  is  that  of  physi¬ 
cal  separation;  the  second  is  that  of  elec¬ 
tronic  integration,  i.e.  two  or  more  physically 
separate,  and  separated,  models  and  simu¬ 
lations  interacting  with  each  other  because 
they  share  a  common  view  of  their  electronic 
environment.  The  output  resulting  from  the 
execution  of  each  model  is  seen,  interpreted 


and  acted  upon  by  the  other(s)  in  real-time. 

The  Distributed  Interactive  Simulation 
(DIS)  environment  is  an  infrastructure  which 
implements  the  concept  of  ADS.  In  the  cur¬ 
rent  DIS  concept,  the  world  is  modeled  as  a 
set  of  entities  that  interact  with  each  other 
by  events  that  they  cause.  Whenever  the  state 
of  an  entity  changes,  such  changes  are  broad¬ 
cast  over  the  communications  media,  seen 
by  all  other  entities,  ignored  by  some  and 
effect  others:  possibly  causing  them  to  re¬ 
spond,  resulting  in  further  interaction  be¬ 
tween  the  entities  within  the  models. 

A  key  task  of  the  DIS  community  in  defin¬ 
ing  and  providing  the  infrastructure  required 
to  combine  individual  simulations  into  a 
seamless  virtual  world  is  the  establishment 
of  a  .series  of  standards  in  the  areas  of: 

•  Interface  definition 

•  Communication 

•  Representation  of  the  environment 

•  Management 

•  Security 

•  Field  instrumentation 

•  Performance  measurement 

The  initial  focus  of  applying  ADS  has  been 
on  training.  In-roads  are  being  made  in  its 
use  for  evaluating  new  concepts  in  combined 
arms  doctrine,  tactics,  and  to  some  extent, 
joint  interoperability.  To  fulfill  the  vision  of 
broader  military  applications  will  require  its 
extension  into  test  and  evaluation;  mission 
rehearsal;  and  research,  development  and 
acquisition  (RD&A). 

The  highest  potential  pay-off,  at  present, 
appears  to  be  in  several  areas  of  RD&A. 

•  Assisting  in  defining  requirements  for 
new  battlefield  systems,  thereby  reducing  the 
risks  of  embarking  on  lengthy  and  expensive 
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development  programs. 

•  Playing  a  key  role  in  conducting  certain 
aspects  of  early  user  testing  and  evaluation, 
as  a  weapon  system  design  matures.  Fre¬ 
quent  involvement  by  the  user  in  a  synthetic 
environment  will  make  the  design-test-fix 
cycle  faster.  The  use  of  simulation  during 
design,  testing  and  prototyping  a  design  fix 
(or  model-test-model)  can  ensure  an  accept¬ 
able  fix  before  major  investment  of  funds  in 
an  actual  build. 

•  Conducting  portions  of  weapon  sys¬ 
tems’  developmental  and  operational  test¬ 
ing  and  evaluation  (DT&E  and  OT&E)  in  a 
synthetic  environment.  As  noted  in  Chapter 
5,  several  aspects  of  DT&E  already  involve 
the  use  of  simulation.  Although  models  and 
simulations  have  been  used  fairly  extensively 
in  support  of  OT&E,  extending  the  use  of 
ADS  to  conduct  OT&E  is  relatively  fertile 
ground.  DoD  seeks  to  apply  a  higher  stan¬ 
dard  of  acceptability  for  models  or  simula¬ 
tions  used  in  support  of  OT&E,  requiring 
that  “special  care  is  necessary  to  ensure  they 
are  credible”.*  Understandably  so,  since 
OT&E  is  often  the  last  line  of  defense 
against  carrying  risks  into  the  next  phase  of 
acquisition.  However,  with  VV&A  for  dis¬ 
tributed  M&S  being  more  complicated  than 
for  monolithic  or  stand-alone  M&S,  estab¬ 
lishing  sufficient  credibility  for  their  use  in 
support  of  OT&E,  will  carry  some  special 
challenges. 

Several  technical  issues  still  limit  the  full  ex¬ 
tension  of  ADS  into  the  acquisition  domain. 

•  Program  managers  (PMs)  intending  to 
construct  models  for  use  in  the  DIS  domain 
should  familiarize  themselves  with  the  is- 
sues^  and  track  their  resolution  over  time. 

•  PMs  should  recognize  the  importance 


of  being  in  control  of  the  manner  in  which 
their  system  is  represented  and  used  on  the 
synthetic  battlefield. 

•  In  the  longer  term,  PMs  electing  not  to 
build  DIS-compliant  simulations,  will  run  the 
risk  of  one  being  developed  and  used  out¬ 
side  their  control. 

•  One  of  the  critical  technical  issues  in¬ 
volved  in  extending  the  use  of  ADS  into  ac¬ 
quisition  remains  that  of  establishing  stan¬ 
dards;  permitting  the  models  and  simula¬ 
tions  to  interact. 

6.6  Standards  for  the  Modeling  and  Simu¬ 
lation  Environments 

Charles  D.  Sullivan  defines  a  standard  as  “a 
category  of  documents  whose  function  is  to 
control  some  aspect  of  human  endeavor."'" 
Standards  are  established  to  overlay  a  cer¬ 
tain  discipline  on  a  process  or  product.  Un¬ 
less  such  discipline  is  established  with  spe¬ 
cific  purposes  in  mind,  control  ends  up  be¬ 
ing  established  for  no  reason  other  than  the 
exercise  of  power.  The  bad  press  DoD’s  ap¬ 
plication  of  standards  has  had  in  recent  years 
is  not  the  result  of  the  existence  of  these  stan¬ 
dards;  rather,  it  is  the  sometimes  blind  im¬ 
position  or  implementation  of  them. 

The  objective  behind  the  creation  of  a  stan¬ 
dard  must  be  considered  from  two,  often 
very  different,  perspectives;  “Why  create  a 
standard?”  and  “Why  use  one?”  The  clearer 
the  objectives  behind  the  creation  of  a  stan¬ 
dard,  the  more  likely  it  is  that  the  standard 
will  be  used  by  those  for  whom  it  is  intended. 

If  there  were  ever  a  discipline  that  could  gain 
significantly  from  implementation  and  en¬ 
forcement  of  a  reasoned  approach  to  estab¬ 
lishment  and  adoption  of  standards,  it  is  that 
of  the  development  of  models. 
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Currently  there  is  no  universal  standard 
within  DoD  for  models  and  simulations. 
This  appears  to  be  a  conscious  decision  to 
force  an  evolution  toward  a  set  of  broad  stan¬ 
dards  that  could  serve  wider  needs  within  the 
acquisition  community.  There  are  standards 
development  efforts  underway  within  the 
various  functional  communities  and  appli¬ 
cation  domains.  Over  time,  the  need  to  effi¬ 
ciently  share  information  between  functional 
areas  will  provide  a  forcing  function  to  evolve 
a  set  of  common  standards.  These  need  to 
be  broad  enough  to  encompass  the  needs  of 
each  community  without  requiring  unaccept¬ 
able  compromises,  yet  definitive  enough  to 
permit  cross-functional  interaction. 

Even  a  model  developed  to  comply  with  a 
certain  set  of  standards  is  at  risk  of  drifting 
over  time,  as  its  configuration  changes.  To 
ensure  the  continuing  ability  of  a  model  to 
serve  the  purpose  it  was  built  for,  its  cred¬ 
ibility  will  have  to  be  maintained  over  its  life 
cycle  through  a  planned  maintenance  and 
re-accreditation  process. 

6.7  Maintaining  Credibility  of  M&S 

Having  established  a  model’s  suitability  for 
use,  whether  for  a  class  of  applications  or 
for  a  specific  one,  a  rigorous  effort  is  re¬ 
quired  to  ensure  that  it  remains  credible. 

Since  a  portion  of  most  models  or  simula¬ 
tions  is  software,  loss  of  configuration  con¬ 
trol  can  occur  insidiously,  over  time,  and  in¬ 
visibly.  An  Army  Audit  Agency  Advisory 
Report  mentions  that,  over  a  period  of  two 
years,  “one  model  was  modified  623  times 
(including  9  major  changes)  without 
revalidation”." 

Managing  and  controlling  a  model’s  configu¬ 
ration  becomes  crucial  in  maintaining  it  as  a 
credible  tool  to  support  analyses. 


6.7.1  Configuration  Management  (CM) 

During  the  process  of  developing  DA  PAM 
5-11,  yV&A  ofAmiy  Models  and  Simulations, 
the  writers  concluded  that  a  strong  CM  plan 
is  one  of  the  critical  ingredients  in  ensuring 
the  continued  credibility  of  models  and  simu¬ 
lations.  The  CM  plan  for  a  model  must  en¬ 
sure  controls  for  the  model  itself,  its  devel¬ 
opment  process  and  the  input  data. 

Configuration  management  is  defined"  as  a 
discipline  applying  technical  and  administra¬ 
tive  direction  over  the  life  cycle  of  an  item  to: 

•  Identify  and  document  the  functional 
and  physical  characteristics  of  the  item  and 
its  major  parts; 

•  Control  changes  to  these  parts  and  to 
their  related  documentation; 

•  Establish  a  process  for  maintaining  sta¬ 
tus  of  proposed  changes,  implementation 
status  of  approved  changes,  etc.;  and 

•  Establish  a  process  for  conduct  of  au¬ 
dits  to  verify  conformance  of  the  item’s  (and 
its  major  parts’)  design  and  perfornlance 
with  requirements  documentation. 

This  general  definition  covers  any  configu¬ 
ration  item  within  a  system,  be  it  hardware, 
software  or  firmware. 

There  are  several  reasons  why  a  good  CM  is 
an  especially  important  activity  for  models 
and  simulations. 

1.  Fadlitates  repeatability:  Maintaining  a 
record  copy  of  an  M&S  used  in  providing 
information  to  .support  an  analysis,  along 
with  its  associated  input  data,  provides  an 
M&S  user  the  ability  to  reconstruct  the 
analysis  at  a  later  date. 
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2.  Enables  traceability:  Maintaining  a 
clear  audit  trail  of  changes  to  a  model  and 
input  data  provides  a  mechanism  for  corre¬ 
lating  each  change  to  the  circumstance(s) 
generating  the  requirement.  This  allows 
proper  analysis  of  the  cause-and-effect  rela¬ 
tionships  among  a  series  of  experiments. 

3  Maintains  credibility:  This  is  probably 
the  strongest  argument  for  a  rigorous  CM 
program,  and  is  directly  linked  to  the  previ¬ 
ous  two  items.  A  well  constructed  and  imple¬ 
mented  CM  plan  for  a  model  will  signifi¬ 
cantly  reduce  the  risk  of  its  data  output  be¬ 
ing  misused  in  an  analysis. 

4.  Maintains  interfaces:  Integrated, 
interoperable  and  interactive  models  and 
simulations  are  becoming  more  important 
to  efficient  conduct  of  DoD’s  business.  It  is 
critical  to  identify  and  control  any  changes 
which  may  disrupt  this  interoperability. 

Three  elements  are  considered  essential 
for  adequate  CM; 

1.  A  CM  policy  establishing  the  admin¬ 
istrative  processes  for  approving  and  docu¬ 
menting  changes  to  the  model  or  simulation; 

2.  A  CM  plan  describing  how  changes  to 
the  existing  configuration  will  be  accom¬ 
plished;  and 

3.  A  CM  board  or  official  with  authority 
to  approve  proposed  changes  to  the  cur¬ 
rently  approved  configuration. 

Some  managers  have  encountered  problems 
following  release  of  a  model  and  recommend 
more  stringent  measures  to  protect  the 
configuration.  The  following  guidelines  have 
proven  effective  for  several  model  managers. 

•  Maintaining  a  baseline  copy  of  each 


version 

•  Tight  control  of  changes  by  users 
(implementing  a  policy  of  not  releasing 
source  code) 

•  Execution  of  various  forms  of  agree¬ 
ments  with  model  users 

•  Formation  of  user’s  groups  that  are  par¬ 
ticularly  useful  in  addressing  CM  issues  per¬ 
taining  to; 

—  Documenting  changes  made  (or 
suggested)  by  users  other  than  the 
model  manager; 

—  Evaluation  of  proposed  changes; 
and 

—  Suggesting  and  developing  meth¬ 
odologies  for  V&V  of  changes. 

Across  the  board,  one  of  the  most  useful 
mechanisms  for  PMs  appears  to  be  mainte¬ 
nance  of  a  record  copy  (baseline  version  plus 
changes)  of  each  model  or  simulation  used, 
along  with  the  validated  data  for  that  version. 
This  is  of  tremendous  benefit  to  the  model 
user  since  analyses  supporting  a  decision 
must  often  be  traceable  to  the  specific  con¬ 
figuration  of  a  model  exercised;  to  provide 
information  for  the  analyses  and  the  data 
used  in  its  execution. 

Any  CM  can  be  resource  intensive.  Managers 
are  advised  to  assess  any  potential  impacts.  The 
PMs  need  to  ask  such  questions  as; 

•  What  resources  are  going  to  be  required 
to  implement  an  effective  CM  program? 

•  Do  program  management  office  per¬ 
sonnel  have  the  required  training  (technical 
and  other)  to  oversee  CM  activities,  facili- 
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tate  user  group  meetings,  etc.? 

•  If  the  model  manager  maintains  total 
control  of  source  code,  does  the  office  have 
the  resources  to  be  the  only  one  making 
changes  to  the  model?  If  not,  will  users  be 
willing  to  provide  reimbursement  for  the 
effort(s)? 

If  the  objective  is  to  get  others  to  use  a  model 
because  of  its  ability  to  improve  a  process  or 
product,  it  does  not  make  much  sen.se  to 
implement  inordinately  stringent  controls. 
However,  there  may  also  be  valid  concerns 


about  proliferation  of  unauthorized  versions, 
loss  of  credibility  in  the  model  as  a  tool,  etc. 
The  structure  of  the  CM  process  should 
implement  a  level  of  control  needed,  with¬ 
out  making  it  unnecessarily  hard  for  poten¬ 
tial  users  to  get  the  model  and  use  it. 

A  good  CM  plan  is  critical  to  maintain  the 
credibility  of  a  model,  ensure  its  continued 
compliance  with  standards  to  which  it  \,as 
designed  and  ensure  its  overall  utility  to  the 
program.  Investment  in  good  M&S  CM  may 
minimize  future  problems  with  credibility. 
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7 

MANAGEMENT  CONSIDERATIONS 
IN  MODELING  AND  SIMULATION 


7.1  Introduction 

This  chapter  will  consolidate  some  of  the 
management  considerations  involved  in  the 
use  of  models  and  simulations  in  systems 
acquisition.  It  will  also  provide  a  notional 
process  to  plan  the  modeling  and  simulation 
(M&S)  effort  for  a  program. 

7.2  Survey  of  ACAT  I  &  II  PMOs 

In  September  1993,  the  Defense  Model¬ 
ing  and  Simulation  Office  (DMSO)  Acqui¬ 
sition  Task  Force  on  Modeling  and  Simula¬ 
tion  (ATFM&S)  initiated  a  survey  of  DoD 
ACAT  I  &  II  Program  Management  Offices 
(PMOs). 

7.2,1  Purpose 

The  purpose  of  the  survey  was  threefold: 

1.  Gain  information  on  how  the  armed 
services  were  currently  using  M&S; 

2.  Determine  the  degree  of  existing 
cross-functionality  in  current  models  and 
simulations;  and 


3.  Identify  improvements  the  DoD  acqui¬ 
sition  community  felt  were  most  needed  to 
provide  better  support  to  acquisition  pro¬ 
grams  in  the  future. 

7.2.2  Survey  method 

The  survey  was  sent  to  selected  program  of¬ 
fices  across  the  Army,  Navy.  Marine  Corps 
and  Air  Force.  Selected  survey  responses 
were  followed-up  through  site  visits  and  in¬ 
terviews  (one-on-one  and  group). 

7.2.3  Summary  of  survey  results 

The  results  of  the  56  responses  provide  some 
valuable  insights  into  M&S  use.  Response 
information  pertaining  to  how  PMOs  are 
currently  using  M&S  is  summarized  in  Fig¬ 
ure  7-1 ';  and  further  explained  below. 

7.2.3.1  Modeling  and  Simulation  Use 

A  majority  of  respondents  indicated  mod- 
erate-to-extensive  use  of  models  and  simu¬ 
lations  to  support  acquisition  decision  pro¬ 
cesses.  While  the  largest  number  of  uses 
were  in  the  threat,  requirements,  cost  and 
testing  areas;  the  largest  single  use  by  pro- 
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How  PMOs  Are  Using  M&S  Today 

(ATFMS  Survey  of  Select  ACAT  I  &  II  PMOs) 
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Figure  7-1.  How  Program  Management  Offices  are  Currently 
Using  Modeling  &  Simulation 


gram  managers  (PMs)  was  in  the  area  of  sys¬ 
tems  engineering  and  performance  evalua¬ 
tion.  Overall,  the  survey  confirmed  that 
models  and  simulations  were  being  used 
across  a  variety  of  functional  areas. 

7.23.2  Modeling  &  Simulation  Needs 

The  sur/ey  results,  with  respect  to  expressed 
needs  of  PMOs,  are  summarized  in  Figure 
7-21  Some  of  the  key  issues  recommended 
for  addressal  were  to  provide  adequate  re¬ 
sources  for  M&S  efforts,  consistency  in  cost 


models,  user-friendly  models  and  simula¬ 
tions,  and  more  and  better  information 
(training  and  reference  material)  on  the  use 
of  models  and  simulations. 

With  respect  to  what  capabilities  the  PMs 
would  like  to  see  improved,  the  majority  of 
responses  were  in  the  areas  of:  improved 
scheduling  models;  improved  cost  and  op¬ 
erational  effectiveness  analysis  (COEA)  and 
life-cycle  cost  (LCC)  tools;  improved 
interopera  lility  with  other  models;  and  im¬ 
proved  program  specific  models. 
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Summary  of  Needs  Perceived  by  PMOs 

(ATFMS  Survey  of  Select  ACAT  I  &  II  PMOs) 
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Figure  7-2.  Summary  of  Modeling  and  Simulation  Needs  Perceived  by  PMOs 


7.2.3.3  Centralized  vs  Decentralized 
Management 

The  survey  indicated  a  predominance  of 
decentralized  management,  wherein  the 
model  user(s)  managed  the  development 
and  use  of  models  they  needed  for  their  par¬ 
ticular  functional  area.  However,  the  survey 
indicated  a  strong  correlation  between  the  type 
of  management  of  M&S  efforts  and  perceived 
overall  contribution  from  their  use. 

The  PMOs  reporting  centralized  manage¬ 


ment  perceived  the  overall  contribution  as 
considerable  to  extensive.  This  relationship 
is  strong  enough  to  warrant  consideration  of 
centrally  managing  the  M&S  effort  within 
the  PMO.  Each  PM  must  make  a  conscious 
choice  in  the  type  of  M&S  management  to 
implement.  Factors  to  consider  in  determin¬ 
ing  whether  to  centralize  management  un¬ 
der  a  simulation  manager  include:  availabil¬ 
ity  of  an  individual  with  the  requisite  breadth 
of  knowledge  and  training;  the  PM’s  man¬ 
agement  style;  the  PM’s  organizational  struc¬ 
ture;  and  the  scope  of  the  effort. 
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The  scope  of  the  M&S  effort  within  a  PMO 
will  be  determined  by  factors  such  as  the 
number  of  models  and  simulations  devel¬ 
oped,  used  and  maintained;  the  purposes  for 
which  they  are  employed;  and  their  complex¬ 
ity  and  sophistication.  A  universal  model 
(one  that  models  every  aspect  of  the  pro¬ 
gram)  for  a  given  program,  is  panacea.  A 
given  program  will  normally  require  the  use 
of  several  models  and  simulations,  desirably 
using  common  data  bases.  Their  manage¬ 
ment  is  not  trivial. 

7.3  Planning  the  Modeling  and  Simulation 
Effort 

A  manager,  when  initiating  the  M&S  plan¬ 
ning  effort,  must  take  a  bottoms-up  ap¬ 
proach.  The  tool  has  to  fit  the  need,  requir¬ 
ing  managers  to  start  determining  their  M&S 
requirements  based  upon  what  they  are  try¬ 
ing  to  accomplish. 

The  decision  to  employ  models  and  simula¬ 
tions,  as  well  as  the  subsequent  decision(s) 
to  use  or  modify  an  existing  model,  or  cre¬ 
ate  a  new  model,  will  depend  on  the  circum¬ 
stances  and  specifics  of  a  given  program.  The 
questions  which  follow  are  not  offered  to 
trivialize  the  process  with  a  cookbook  solu¬ 
tion. 

The  templates  in  Appendix  G  provide  a 
baseline  from  which  PMs  can  develop  ques¬ 
tions  reflecting  their  programs.  The  support¬ 
ing  functional  managers  can  then  step 
through  the  questions  below  to  fill  in  the 
blanks. 

•  What  am  I  trying  to  achieve?  What’s 
the  objective?  What  question(s)  am  I  trying 
to  answer? 

•  What  analyses  will  have  to  be  con¬ 
ducted?  When  will  the  results  be  needed? 


How  long  will  the  analyses  take?  Who  will 
do  them? 

•  What  information  is  required  to  sup¬ 
port  the  analyses?  How  accurate  does  the 
information  have  to  be? 

•  What’s  the  most  efficient  way  to  get  the 
information?  Are  several  excursions  (or  it¬ 
erations)  going  to  be  required?  Is  it  a  one¬ 
time  requirement,  or  will  this  be  an  on-go¬ 
ing  requirement?  Do  I  need  a  model  to  pro¬ 
vide  the  information? 

•  Can  any  existing  models  or  simulations 
provide  the  information  I  need?  What  is  the 
verification,  validation  and  accreditation 
(VV&A)  status?  Are  they  accredited  for  my 
class  of  system?  Will  they  need  modification? 
What’s  the  extent  of  modification?  Can  the 
model  owner(s)  do  the  modification  in- 
house?  Can  I?  Any  proprietary  issues  that 
may  lock  me  in  to  a  sole  source? 

•  What  data  are  required  by  these 
model(s)?  Where  and  how  can  the  data  be 
obtained? 

•  What  resources  (funds,  people,  time, 
test  articles,  hardware,  software,  range  fa¬ 
cilities,  documentation)  will  1  need  to: 

—  build  or  modify  the  model(s)? 

—  conduct  VV&A  on  the  model(s)  or 

modification? 

—  implement  configuration  manage¬ 
ment  (CM)? 

—  obtain  and  validate  the  data? 

—  run  the  modei(s)? 

—  analyze  the  output? 


—  ensure  that  the  model(s)  are  main¬ 
tained  to  accurately  represent  my  system 
or  program? 

—  transition  the  models  and  simula¬ 
tions  to  a  supporting  activity  for  mainte¬ 
nance  upon  dissolution  of  my  PMO?  Does 
“operation  and  support’’  funding  provide 
for  model(s)  maintenance  after  transi¬ 
tion? 

•  Does  the  system  design  accommodate 
plans  for  hardware/software-in-the-loop 
(HW/SWIL)  with  regards  to  test  and  instru¬ 
mentation  ports,  etc.  What  do  I  need  to 
populate  my  test  bed(s)  or  simulation 
facility(ies)? 


Are  my  models  and  simulations  consistent 
and  integrated  with  the  rest  of  my  program? 
Are  they  reflected  as  tools  contributing  to  re- 
quirements  verification  in  a  requirements 
correlation  matrix  (RCM)?  Are  their  char¬ 
acteristics  consistent  with  the  COEA,  test  and 
evaluation  masterplan  (TEMP),  operational 
requirements  document  (ORD),  and  acqui¬ 
sition  program  baseline  (APB)  with  respect 
to  measures  of  outcome,  effectiveness  and 
performance  (MOOs,  MOEs,  and  MOPs)? 


Once  managers  have  completed  this  thought 
process,  all  the  ingredients  of  a  plan  are  in 
place. 

•  The  tasks,  functions  or  decisions  to  be 
supported  by  M&S 

•  The  specific  M&S  tools  required 

•  When  they  are  needed 

•  How  the  M&S  are  going  to  be  acquired 


•  The  re.sources  required  to  acquire  and 
manage  the  M&S 

7.3.1  The  Simulation  Support  Plan 

In  an  attempt  to  ensure  early  consideration 
and  planning  the  use  of  M&S  in  major  pro¬ 
grams,  the  Army  has  mandated  development 
of  a  Simulation  Support  Plan  (SSP)  for  all 
ACAT  I  and  ACAT  II  programs,  as  well  as 
for  Advanced  Technology  Demonstrations 
(ATDs).  The  SSP  forces  PMs  to  view  the 
entire  program  in  the  context  of  the  deci¬ 
sions  to  be  made,  timing,  and  impact  (rela¬ 
tive  importance).  It  forces  managers  to  con¬ 
sider  information  needs  in  light  of  the  deci¬ 
sions  to  be  supported,  and  assess  the  appli¬ 
cability  of  models  and  simulation  to  provid¬ 
ing  the  information.  Figure  7-3  is  a  proposed 
outline  for  an  SSP. 

PMs  must  consider  the  resources  required 
to  build  the  program,  which  includes  the  SSP. 
The  SSP  is  considered  an  evolutionary  docu¬ 
ment,  and  is  supposed  to  be  refined,  through 
periodic  review,  as  the  program  progresses. 
Like  other  components  of  an  acquisition 
program,  the  M&S  requirements  will  coa¬ 
lesce  and  get  more  detailed  over  time. 


It  is  not  the  SSP  itself,  but  the  “journey” 
through  the  process  of  identifying  the  program 's 
M&S  needs  that  is  more  valuable.  Creating  a 
bureaucracy  that  simply  requires  “another 
plan”,  would  be  counter-productive. 


One  of  the  initial  challenges  a  manager  will 
face  is  in  trying  to  identify  existing  resources 
that  could  be  used  (either  as  is  or  with  modi¬ 
fications)  to  address  the  M&S  needs.  A 
plethora  of  models  and  simulations  have  al¬ 
ready  been  developed  and  are  in  various 
stages  of  accreditation  for  different  pur- 
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Outline  for  a  Simulation  Support  Plan  (SSP) 

I.  Purpose 

•  Brief  statement  as  to  why  plan  is  required 

-  Focus  on  the  use  of  M&S  in  the  program 

II.  Executive  Summary 

•  Summary  narrative  of  Section  V. 

III.  System  Description 

•  Brief  summary  of  weapon  system 

IV.  Program  Acquisition  Strategy 

•  Brief  synopsis  of  system  acquisition  strategy 

•  Overview  of  M&S  acquisition  strategy 

-  Include  role  of  weapon  system  M&S  in  the  distributed  environment 

V.  Simulation  Approach/Strategy  and  Rationale 

•  What  M&S  is  being  done,  and  why 

A.  M&S  used  to  date 

•  Discuss  all  previous  M&S  used  to  support  the  program 

-  Name/Type  of  M&S  (Live,  Constructive  or  Virtual) 

•  V&V  performed  on  M&S 

•  Accreditation  status  of  M&S 

•  To  what  phase/miiestone  was  M&S  applied 

-  Issues  addressed  and  results 

•  Include  M&S  supporting: 

-  Mission  area  analyses 

-  Operational  analyses 

-  Requirements  trade-offs 

-  Conceptual  design  studies 

-  Systems  engineering  trade-offs 

-  Cost  and  operation  effectiveness  analyses 

-  Logistics  analyses 

-  Test  and  evaluation 

-  Training 

B.  Future  Simulation 

•  Include  on-going  M&S 

•  Ail  planned  M&S  for  future  milestones 

•  How  planned  M&S  will  support  future  milestones 

•  How  planned  M&S  supports  the  Service’s  vision  for  M&S 

(continued) 


Figure  7-3.  Outline  for  a  Simulation  Support  Plan  (SSP) 
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Outline  for  a  Simulation  Support  Plan  (SSP)  (continued) 

VI.  Related  Simulation  Activities 
Include 

•  Other  M&S  activities  the  system  relies  upon 

•  Other  systems  that  rely  upon  this  system’s  M&S  tools 

•  All  other  related  M&S  that  affect  this  system 

Vli.  Management 

•  Provide  wiring  diagram  of  PMO 

•  Show  simulation  manager  (if  assigned)  in  diagram 

•  Describe  how  simulation  manager  interacts  with  acquisition  community 

VIII.  Facilities/Equipment  Requirements 
Describe  facility  requirements  for  all  M&S 

•  Ail  facilities,  hardware,  software,  data,  etc. 

-  Provided  by  PM,  other  Gov’t  activities,  contractor's) 

-  identify  who  will  provide 

-  Identify  schedule  requirements  and  availability  of  items  to  support 
schedule 

Ensure  government  ownership  of  equipment  (Including  simulators, 
hardware,  software,  data,  etc.)  critical  for  cost  effective  government 
management  of  M&S 

IX.  Funding 

•  Outline  all  expected  expenditures  to  support  M&S  program 

•  Include  funded  and  unfunded 

•  Designate  type  of  funding  (by  Program  Elements  (PE),  project,  etc.) 

X.  Remarks/Supplemental  Information 

•  Any  comments  or  related  information 

XI.  Appendices 

•  Program  Schedules 

•  M&S  Schedules 

•  Acronyms  and  abbreviations 

•  Related  standards 

•  Related  government  documents 


Figure  7-3  (Cont’d).  Outline  for  a  Simulation  Support  Plan  (SSP) 
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poses.  However,  the  PM  must  know  how  to 
get  the  information  needed  to  make  a  deci¬ 
sion  as  to  whether  one  or  more  of  these  could 
satisfy  an  M&S  need. 

7.3.2  Models  and  Simulations  Catalogs 

The  discipline  of  modeling  has  been  likened 
to  a  cottage  industry,  with  a  proliferation  of 
models  among  the  various  Services,  and  even 
within  individual  services.  A  March  1993 
DoD  Inspector  General’s  (IG’s)  audit  of  se¬ 
lected  models  and  simulations  and  their  de¬ 
veloping  and  using  activities  concluded  that 
“Model  and  simulation  projects  are  being 
procured  and  developed  within  the  DoD 
without  adequate  coordination  and  control”, 
and  that  “This  has  resulted  in  redundant 
models/simulations  and  a  proliferation  of 
system  architectures  and  libraries”.' 

The  IG  reports:  results  can  be  contributed 
to  lack  of  communication.  One  Service  not 
being  aware  of  what  models  the  others  have. 
This  can  also  be  extended  to  the  Services  lack 
of  awareness  of  the  applicability,  modifiabil¬ 
ity,  portability,  or  V&  V  status  of  those  mod¬ 
els  within  their  respective  Service.  This  is¬ 
sue  is  being  addressed,  to  some  extent,  by 
the  creation  of  models  and  simulations  cata¬ 
logs  by  each  Service  and  the  Joint  Staff. 
These  efforts,  while  valuable,  have  not  re¬ 
sulted  in  complete  listings.  Many  model  us¬ 
ers  are  unaware  of  the  existence  of  these 
catalogs;  many  others,  though  aware  of  their 
existence,  may  not  have  submitted  their 
models  for  inclusion. 

Appendices  A  through  E  include  reference 
to  the  service  catalog(s)  and  information 
on  how  to  gain  access  to  the  information. 
Appendix  F  includes  a  partial  list  of  other 
sources  of  information.  On-line  access  to 
many  catalogs  is  also  provided  by  the  Ser¬ 
vice  sponsor,  and  by  the  DMSO,  through 


their  Modeling  and  Simulation  Information 
System  (MSIS). 

Model  developers  and  users  should  use  the 
catalogs  as  a  starting  point  while  preparing 
an  SSP.  However,  they  are  not  all-inclusive; 
so,  even  if  they  do  not  list  a  model  that  satis¬ 
fies  the  need,  points  of  contact  (POCs)  for 
similar  models  may  be  able  to  provide  some 
additional  leads. 

The  PMOs  often  do  not  have  resident  ex¬ 
perts  to  answer  the  questions  required  to 
plan  their  use  of  M&S.  Often  times  DoD 
PMOs  must  turn  to  contractors,  consultants 
or  other  agencies  for  assistance  in  determin¬ 
ing  the  M&S  needs  for  the  program,  as  well 
as  developing  the  M&S  tools. 

733  Selecting  a  Modeling  and  Simulations 
Developer 

This  section  will  assist  the  PM  in  identify¬ 
ing  or  selecting  a  contractor  to  assist  in 
the  M&S  effort. 

The  PM  must  identify  a  M&S  developer. 
This  could  be  a  government  agency,  an  in¬ 
dependent  contractor  or  the  prime  system 
contractor.  Obviously,  the  developer  must 
understand  M&S  and  the  PM’s  unique  re¬ 
quirements.  A  list  of  questions  that  may  help 
the  PM  select  a  developer  is  provided  by  Van 
B.  Norman."  This  list  is  based  upon  twenty- 
years  of  building  simulation  models,  and  hir¬ 
ing  and  managing  simulation  consultants. 

In  DoD’s  case,  these  questions  can  form  the 
basis  for  part(s)  of  a  Request  For  Proposal 
(RFP),  as  well  as  provide  ingredients  for  es¬ 
tablishing  Source  Selection  Criteria. 

The  specific  questions,  and  the  level  to  which 
they  will  have  to  be  pursued,  will  depend 
upon  certain  factors  within  the  program  ac- 
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quisition  strategy.  Among  these  are  factors 
such  as:  whether  the  main  effort  and  the 
simulation  effort  are  under  a  common  con¬ 
tract;  who  is  integrating  the  simulation  ef¬ 
fort;  whether  the  system  and  simulation  de¬ 
velopment  efforts  are  complementary,  with 
each  leveraging  off  the  other;  or  whether 
they  are  independent. 

a.  What  is  the  contractor’s  experience  with 
this  type  of  system?  What  is  the  contractor’s 
track  record? 

A  contractor’s  experience  with  similar  sys¬ 
tems  is  important,  since  it  normally  gener¬ 
ates  efficiency,  but  it  is  not  essential.  If  a  spe¬ 
cific  contractor  has  an  unproven  record,  a 
software  capability  evaluation^  may  provide 
some  insight  into  their  ability  to  take  on  a 
complex  software  modeling  effort.  This  is 
particularly  true  if  the  M&S  development  is 
a  parallel  effort  to  the  system  development. 
The  evaluation  augments  the  acquisition 
process  by  determining  a  contractor’s 
strengths  and  weaknesses  with  respect  to  a 
maturity  model.  It  establishes  a  method  for 
comparing  a  contractor’s  “software  capabil¬ 
ity”  against  a  standard  set  of  criteria  as  a 
source  selection  criterion. 

b.  How  will  the  contractor  approach  the 
construction  of  the  model?  What  simulation 
software  will  be  used? 

Maybe  there  is  a  requirement  for  a  specific 
software  tool.  Ada  may  not  be  the  most  ap¬ 
propriate  —  if  possible,  choose  a  simulation 
tool  that  is  widely  used  and  will  be  around 
for  a  few  years. 

c.  Will  the  contractor  produce  a  written 
specification  describing  the  system  to  be  mod¬ 
eled,  including  all  assumptions  and  questions 
to  be  answered?  Are  you  going  to  be  providing 
the  contractor  a  specification? 


In  any  case,  the  specification  is  necessary  to 
ensure  that  everyone  is  working  toward  the 
same  goal. 

d.  What  questions  about  the  system  can¬ 
not  be  answered  from  use  of  the  model? 

Models  need  to  be  constructed  with  certain 
questions  in  mind.  An  understanding  of  what 
the  model  will  not  answer  is  crucial  to  pre¬ 
vent  misunderstanding  between  the  project 
office  and  the  developing  contractor.  Another 
reason  why  the  model  specification  is  so 
important. 

e.  What  is  the  development  schedule? 

The  model  or  simulation  supports  certain 
information  needs  of  the  project  office. 
Unless  this  information  is  timely,  it  could  be 
worthless.  The  contractor’s  prior  record  with 
respect  to  ability  to  deliver  on  schedule 
should  be  a  important  criterion  in  selection. 

f.  How  did  the  contractor  arrive  at  the  cost 
estimate  for  the  projects? 

Regardless  of  whether  the  contractor  is 
working  on  a  cost-plus  or  fixed  price  basis, 
the  contractor  must  understand  the  scope  of 
work  and  schedule  to  develop  a  credible  cost 
estimate. 

g.  How  do  I  determine  value  for  cost? 

Norman  likens  simulation  consulting  to 
brain  surgery — “if  you  want  the  lowest  priced 
surgeon  opening  your  head,  then  good 
luck”,*  he  says.  He  emphasizes  the  need  to 
know  the  experience,  expertise,  and  record 
of  each  candidate  contractor,  and  to  balance 
these  against  the  price  being  charged  to  de¬ 
termine  value. 

h.  What  data  are  required  for  the  model? 


Norman  contends  that  most  contractors 
won’t  know  your  business  well  enough  to 
collect  the  required  data.  The  government 
will  often  need  to  provide  the  data.  Fortu¬ 
nately,  sources  of  valid  data  exist  within  each 
Service  and  DoD.  Each  Service’s  M&S  of¬ 
fice  can  provide  authoritative  sources  of 
data.  Despite  the  unique  needs  of  each  Ser¬ 
vice,  a  common  dictionary  of  data  is  required 
among  the  Services. 

i.  Who  will  collect  the  data?  When  will  it 
be  needed?  What  format  will  it  be  needed? 
Who  will  certify  the  data? 

Crucial  questions,  if  the  government  is  go¬ 
ing  to  be  on  the  hook  to  provide  these  to  the 
modeler.  In  fact,  any  potential  disconnect 
between  the  contractor’s  requirements  for 
data  and  the  government’s  ability  to  provide 
it  must  be  worked  out  early.  The  PMs  must 
also  understand  the  resource  implications  of 
data  collection. 

j .  What  parts  of  the  system  will  be  detailed 
and  what  parts  will  be  simplified? 

Again,  an  issue  that  the  contractor  must  ad¬ 
dress  in  the  proposed  specification. 

k.  What  types  of  model  experiments  will 
be  run? 

As  the  model  is  built,  experimentation  pro¬ 
vides  answers.  If  the  modeler  knows  what 
types  of  experiments  are  contemplated,  the 
model  can  be  built  to  make  the  experiments 
easier. 

l.  How  much  time  will  be  allowed  for  ex¬ 
perimentation? 

The  user’s  understanding  of  the  system  may 
change  after  reviewing  experiment  results, 
and  the  scope  of  work  may  be  impacted.  A 


well  structured  CM  process  will  make  this 
less  painful. 

m.  How  will  you  be  assured  the  model  is 
“correct”? 

The  government  must  be  an  active  player  in 
the  V&V  process.  The  PMs  must  also  be 
mindful  of  the  accreditation  authority’s  re¬ 
quirements  from  the  start — the  require¬ 
ments  must  be  built  into  the  contract. 

n.  What  is  the  schedule  for  periodic  model 
review  meetings? 

This  is  a  crucial  management  mechanism  for 
ensuring  that  incremental  model  develop¬ 
ment  is  on  track  from  cost,  schedule  and 
performance  viewpoints. 

o.  Can  you  use  the  model  internally  after 
the  contractor  is  done? 

The  model  has  long  term  value.  The  system 
will  probably  change  over  time  and  the 
model  must  be  modified.  This  relates  back 
to  the  need  for  CM  and  the  requirement  for 
adequate  documentation. 

Reusability  of  software  and  the  increasing 
move  toward  reconfigurable  simulators  and 
simulations  make  this  even  more  important. 
Also,  the  government’s  aversion  to  locking 
itself  into  a  developer  drives  the  need  for 
the  government  to  identify  in  the  contract 
all  its  MifeS  deliverable  requirements,  and 
timing.  Inclusion  of  contractor  proprietary 
material  or  data,  without  adequate  rights 
being  released  to  (procured  by)  the  govern¬ 
ment,  could  lead  to  problems. 

p.  What  could  go  wrong  with  this  part  of 
the  project? 

Monitor  the  model  (or  simulation)  develop- 
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ment  effort  as  it  proceeds.  Given  the  crite¬ 
ria  fur  establishment  of  Work  Breakdown 
Structure  (WBS)  elements  (MIL-STD 
881B),  and  Configuration  Items  (MIL-STD 
973),  PMs  must: 

—  Ensure  visibility  of  the  M&S  efforts 
at  the  appropriate  level  for  management 
and 

—  Incorporate  the  development  efforts 
into  their  risk  management  program. 

q.  What  kinds  of  analyses  will  the  contrac¬ 
tor  petform,  such  as  confidence  interval  cal¬ 
culation  and  design  of  experiments? 

Whether  it  is  the  contractor  or  someone  else 
who  is  going  to  be  performing  an  analysis 
using  the  model’s  output,  the  analyst’s  re¬ 
quirements  have  to  be  considered  in  design¬ 
ing  the  model. 

Norman  rounds  out  his  suite  of  questions 
ensuring  that  the  contractor  will  assist  in 
gaining  management  and  team  support  for 
the  model  and  for  its  use  in  support  of  the 
analyses  at  hand.  The  contractor  is,  after  ail, 
part  of  the  team. 

r.  “How  will  the  contractor  assist  in  ex¬ 
plaining  the  benefits  and  limitations  of  the 
model?”  “Will  the  contractor  assist  in  present¬ 
ing  the  model  results  to  management  (deci¬ 
sion  makers)?”  and  “Does  the  contractor  have 
the  capability  to  provide  a  video  of  the  model’s 
animation?” 


All  of  these  will  go  far  in  gaining  and  main¬ 
taining  support  for  the  effort,  since  they  cre¬ 
ate  an  understanding  of  the  need  for  the 
model  and  subsequent  analyses  of  its  results. 

7.4  Models  and  Simulations  as  Deliverables 

The  PM  must  be  aware  that  some  models 
and  simulations  will  be  developed  by  the 
prime  contractor  as  a  natural  by-product  of 
the  system  design  and  development  process. 
However,  the  capabilities  and  limitations  of 
these  models  and  simulations,  with  respect 
to  the  acquisition  process,  must  be  under¬ 
stood.  Decisions  regarding  whether  or  not 
to  require  specific  M&S  as  deliverables,  must 
be  made  on  a  case -by-case  basis  with  this 
understanding. 

Contractors  may  also  be  reluctant  to  share  key 
algorithms  included  in  simulations  specified 
for  delivery.  Based  on  the  program’s  acquisi¬ 
tion  strategy,  the  PM  must  assess  impacts  of 
any  restrictions  the  contractor  may  include, 
and  determine  whether  (and  how  much)  it 
would  be  worth  paying  for  their  removal. 

A  PM  must  also  recognize  that  when  pro¬ 
duction  is  complete  and  a  contract  ends, 
M&S  support  will  still  be  required  for  the 
remainder  of  the  system’s  life  cycle.  Models 
and  simulations  that  were  constructed  dur¬ 
ing  earlier  phases  of  the  acquisition  process, 
and  refined  as  the  system  evolved,  will  play 
a  major  role  in  evaluating  system  modifica¬ 
tion  alternatives. 
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8 

MOVEMENT  TOWARD 
A  FUTURE  STATE  OF 
MODELING  AND  SIMULATION 


8.1  Introduction 

This  chapter  presents  the  reader  with  a  vi¬ 
sion  of  what  the  combined  technica]  and 
management  state-of-the-art  appear  to  be 
converging  toward.  The  chapter  also  pro¬ 
vides  some  examples  of  DoD  programs: 
where  the  use  of  models  and  simulations  is 
paying  dividends  today. 

8.2  The  Evolution  of  Modeling  and  Simu¬ 
lation  (M&S)  in  Acquisition 

Technology  is  rapidly  evolving  toward,  with 
some  contending  that  it  has  already  arrived 
at,  a  state  permitting  the  increasingly  sophis¬ 
ticated  implementation  of  integrated  prod¬ 
uct  and  process  teams.  Acquisition  manage¬ 
ment  will  need  to  evolve  in  directions  that 
will  allow  managers  to  take  advantage  of  this 
integration. 

A  migration  is  occurring  from  predominant 
use  of  live  and  constructive  simulations,  to 
increased  interest  on  the  use  of  virtual  simu¬ 
lations.  This  shift  is  supported  by  rapid  im¬ 


provements  in  the  sophistication  of  informa¬ 
tion  processing  and  display  technologies. 
However,  today’s  technical  and  managerial 
use  of  M&S  in  support  of  systems  acqu^i- 
tion  is  largely  characterized  by  use  of  these 
tools  in  stand-alone  and  system-specific 
modes. 

Scant  information  is  provided  on  existing 
resources,  institutional  barriers  and  emer¬ 
gent  policy.  An  interest  in  increasing  com¬ 
munication  among  functional  areas  observed 
throughout  the  acquisition  community  is 
coupled  with  the  continuing  revolution  in 
information  processing  technologies. 

This  strongly  indicates  that  the  future  state 
of  M&S  in  acquisition  will  consist  of  envi¬ 
ronments  which  seamlessly  integrate  simu¬ 
lations.  Integration  will  occur  among  simu¬ 
lations  of  similar  and  different  classes  (live, 
constructive  and  virtual)  and  across  levels  of 
the  M&S  hierarchy  (engineering,  engage¬ 
ment,  mission/battle  and  theater/campaign); 
while  providing  information  that  will  support 
planning  and  decision  making  in  all  func- 
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tional  areas  and  at  the  requisite  level  of  reso¬ 
lution  for  specific  decisions. 

8.2.1  Vision  for  the  future  state  of  Model¬ 
ing  and  Simulation 

John  Hartley  writes  that  technical  develop¬ 
ments  and  environmental  changes  “will  con¬ 
tinue  to  move  the  goal  posts”.'  More  pow¬ 
erful  computer-aided  design  (CAD)  work¬ 
stations,  along  with  distributed  systems  shar¬ 
ing  common  object-oriented  relational  data 
bases,  will  permit  all  departments  working 
on  a  project  access  to  the  same  product  data. 
Product  data,  when  used  by  the  manufactur¬ 
ing  department,  will  be  suitable  for  generat¬ 
ing  machine  tool  paths  and  basic  dimensions 
for  fixtures  or  die  forms. 

Using  the  same  data,  logistics  engineers 
would  be  able  to  conduct  reliability  and 
maintainability  analyses,  and  find  the  loca¬ 
tion  of  key  maintenance  points  such  as  ac¬ 
cess  plates,  sockets,  etc..  The  same  data 
would  be  available  to  cost  estimators  in  such 
a  way  that  costing  could  be  done  easily  and 
fiscal  impacts  of  engineering  design  changes 
would  become  apparent. 

Thus,  each  department  would  see  the  same 
(and  latest)  data  at  all  times,  from  concept 
through  final  design.  Members  of  the  prod¬ 
uct  team,  contacted  about  a  proposed 
change,  would  be  able  to  run  simulations  to 
assess  impacts  from  their  department’s  per¬ 
spectives  and  offer  alternatives,  which  again 
other  members  of  the  team  could  evaluate 
for  impacts. 

Hartley  also  writes  about  the  likelihood  that 
contractors  will  be  brought  more  closely  into 
the  net,  relying  on  the  same  data  base  as  the 
customer.  He  recognizes  similar  concerns  to 
those  voiced  by  defense  contractors;  with 
respect  to  proprietary  nature  of  selected  data 


and  the  resultant  issues  that  DoD  and  con¬ 
tractors  wrestle  with  in  the  Continuous  Ac¬ 
quisition  and  Life-Cycle  Support  (CALS) 
initiative. 

A  close  representation  of  the  future  state  is 
provided  by  the  recent  development  of  tools 
such  as  the  Computer-Aided  Three  Dimen¬ 
sional  Interactive  Applications  (CATIA) 
software  package.  The  CATIA  is  a  series  of 
application  programs  that  interact  to  form  a 
highly  integrated  design,  analysis  and  manu¬ 
facturing  system.-  Boeing  used  CATIA  in 
the  highly  publicized  design  of  the  777  air¬ 
craft,  and  is  currently  using  the  tool  in  the 
design  of  the  F-22  aircraft. 

8.3  Virtual  Prototyping  at  the  US  Army 
Tank  Automotive  and  Armament  Research, 
Development,  and  Engineering  Center 
(TARDEC) 

Engineers  at  TARDEC  are  well  on  their  way 
to  designing  a  limited  implementation  of  the 
vision  for  the  use  of  M&S  in  an  integrated 
acquisition  environment. 

The  TARDEC  engineers  have  developed 
their  virtual  prototype  process  to  lend  itself 
to  continuous  user  participation. (Soldiers 
who  will  eventually  use  the  system  in  the 
field.)  This  process  provides  more  rapid 
feedback  to  the  developer  (the  government- 
contractor  team  responsible  for  developing 
the  system).  Using  advanced  computer 
simulation  enables  early  evaluation  of  new 
vehicle  concepts  without  actually  building  a 
physical  vehicle.  User/developer  agreement 
is  maintained  throughout  the  process  de¬ 
picted  in  Figure  8-1. 

The  steps  in  the  process  are  explained  below. 

Step  1 :  Concepts  -  Solid  models  of  alter¬ 
native  materiel  concepts,  such  as  external 
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•  •  ADVANCING  TECHNOLOGIES  •  • 
STATE  OF  THE  ART  COMPONENTS 


Figure  8-1 .  Virtual  Prototyping  Process 
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The  Tracked  Vehicle  Workstation 
Expanding  Vehicle  Simulation’s  Role 
in  the  Vehicle  Development  Process 
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Figure  8-2.  The  Tracked  Vehicle  Workstation: 

Expanding  Vehicle  Simulation’s  Role  In  the  Vehicle  Development  Process 
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versus  turreted  vehicles  and  wheeled  versus 
tracked  are  developed  to  meet  requirements 
from  the  user  community.  Two-dimensional 
drawings  of  the  conceptual  vehicles  can  be 
produced  from  a  CAD  station. 

Step  2:  Petfonnance  Modeling  -  Analyti¬ 
cal  models  applied  to  the  solid  model  per¬ 
mit  evaluation  of  mobility,  stealth,  survivabil¬ 
ity,  vulnerability,  lethality  and  vehicle  dy¬ 
namics.  Results  of  the  analyses  are  reflected 
through  changes  to  the  solid  model,  optimiz¬ 
ing  design  through  an  iterative  process. 
Competing  conceptual  designs  can  be  evalu¬ 
ated  and  trade-offs  to  satisfy  conflicting  re¬ 
quirements  can  be  worked  with  the  user. 

Step  3:  IVargame  Modeling  -  Resulting 
conceptual  vehicles  are  evaluated  using  initial, 
then  more  detailed  M&S.  Concept  effective¬ 
ness  screening  is  done  using  constructive 
models,  such  as  GROUNDWARS  and 
CASTFOREM. 

Step  4;  Virtual  Mockup! Detailed  Design  - 
Using  concurrent  engineering,  the  selected 
concept  goes  into  detailed  design;  the  solid 
model  is  refined  to  incorporate  actual 
components  using  the  Tracked  Vehicle 
Workstation  as  depicted  in  Figure  8-2.  An 
engineer  selects  components  from,  a  CAD 
parts  data  base,  and  assembles  the  concept 
vehicle.  Simulated  test  scenario(s)  are  de¬ 
fined  using  a  data  base  constructed  from 
previous  instrumented  tests  and  extrapola¬ 
tions. 

A  test  of  the  CAD  model  is  run  in  the  vir¬ 
tual  environment,  with  real-time  quick-look 
animation  available  for  monitoring.  Test  re¬ 
sults  are  analyzed  and,  depending  on  per¬ 
formance.  the  CAD  model  of  the  concept 
vehicle  is  modified.  A  3-D  virtual  prototype 
is  developed  in  which  the  user  can  actually 
explore  the  inside  of  the  vehicle  and  provide 


valuable  human-machine  interface  feedback. 

With  virtual  prototyping,  an  obvious  progres¬ 
sion  would  be  to  allow  the  user  to  fight  the 
vehicle,  by  interfacing  it  with  other  construc¬ 
tive  and  virtual  simulations,  before  construc¬ 
tion  of  a  crew  station  envelope  (step  6). 

Step  5:  Virtual  Factory  -  Actual  manufac¬ 
turing  processes  are  engineered  in  parallel  with 
the  detailed  design  (steps  4).  Machine  tool 
paths,  production  line  set-up.  material  flows, 
assembly,  etc.  are  laid  out  and  .simulated  prior 
to  implementation  on  the  factory  floor. 

Step  6:  Crew  Station  Development  -  The 
virtual  mockup  results  in  the  design  of  a  crew 
station  envelope  (simulator)  to  assess  hu¬ 
man-machine  interfaces  under  static  and 
dynamic  conditions.  The  simulator  is  con¬ 
nected  via  the  distributed  simulation  internet 
to  enable  users  to  evaluate  the  effect  of  the 
concept’s  design  or  tactics  and  force  effec¬ 
tiveness. 

Step  7:  Build  and  Test  Hardware  -  Fabri¬ 
cation  of  a  test-bed  vehicle  is  done  in  part, 
by  directly  outputting  from  the  CAD  station 
to  numerically-controlled  machines.  The 
TARDEC’s  strategy  for  electronics  integra¬ 
tion  is  to  use  a  laboratory  hot-bench  to  re¬ 
solve  integration  issues  prior  to  building  the 
actual  electronics  hardware. 

Step  8:  Manufacturing  -  The  virtual  fac¬ 
tory  provides  the  detailed  layout  for  the  manu¬ 
facturing  process.  Any  changes  in  design,  re¬ 
sulting  from  test-bed  evaluation,  are  fed  back 
into  the  system  to  ensure  compatibility  with 
production  line  set-up,  material  flow,  etc. 

Production  is  based  on  CAD  and  computer- 
aided  manufacturing  (CAM)  machine  out¬ 
put  to  machines,  providing  for  more  accu¬ 
rate  and  faster  parts  manufacture.  The  goal 
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electronically  transfer  parts’  design  directly 
to  a  flexible  manufacturing  facility  for  pro¬ 
duction,  in  the  spirit  of  CALS  implementa¬ 
tion. 

Step  9:  Field  Support  -  The  virtual  proto¬ 
type  process  provides  more  responsive  field 
support  in  terms  of  failure  prediction,  analy¬ 
sis  and  retention  of  the  historical  engineer¬ 
ing  data  base.  Electronic  data  transfer  and 
flexible  manufacturing  also  provide  for  ex¬ 
panded  potential  of  a  smaller  industrial 
base;  possibly  reducing  logistics  spares 
warehousing. 

Not  all  aspects  of  TARDEC's  virtual  proto¬ 
type  process  exist  today.  However,  the  orga¬ 
nization  is  making  significant  strides  to  clos¬ 
ing  the  gaps. 

8.4  Getting  to  the  future  state  of  M &S 

Many  of  the  enabling  technologies,  see  Fig¬ 


ure  8-3,^  associated  with  emerging  M&S  and 
related  tools  that  have  the  potential  to  con¬ 
tribute  to  the  acquisition  process,  are  com¬ 
mercially  driven.  While  this  allows  DoD  to 
leverage  advances  made  in  the  commercial 
market  for  these  technologies,  others  are  of 
specific  interest  to  the  military.  Development 
of  these  latter  technologies  will  be  deter¬ 
mined  by  DoD’s  ability  to  marshal  industry 
innovation  in  the  direction  of  its  interests. 

Each  of  the  enabling  technologies  can  be 
assigned  to  a  level  of  the  notional  hierarchy* 
of  technologies  shown  in  Figure  8-4. 

At  the  base  of  the  hierarchy  are  the  stan¬ 
dards  and  protocols  employed  within  the 
environment.  These  include  many  of  the 
standards  associated  with  modern  software 
systems,  the  exchange  of  product  model  data 
and  simulation  interoperability  standards. 

At  the  next  level  are  the  underlying  collabo- 


Enabling  Technologies 


Commercially  Driven 

Data  base  management  systems 
Man-machine  interfaces 
Software  engineering  tools 
Local  and  wide  area  networks 
High  Performance  computers 
Computer  image  generators 
Microcomputer  systems 
Microprocessors 
Memory 
Mass  storage 
Display  devices 


DoD  Driven 

Manufacturing  Process  Simulations 

Engineering  design  models 

Manned  simulators 

Stochastic  wargaming  simulations 

Semiautomated  forces 

Instrumented  ranges 

Instrumentation 

Simulation  construction  tools 

Multilevel  security 

DoD  protocols 

DoD  data  bases 


Figure  8-3.  Enabling  Technologies 


8-6 


Notional  Hierarchy  of  Technologies 


Figure  8-4.  Notional  Hierarchy  of  Technologies 


rative  technologies  under  development  in 
academia  and  industry.  These  include  efforts 
to  establish  shared  electronic  workspaces, 
develop  customized  software  wrappers  that 
facilitate  the  reuse  of  legacy  code  and  to  cre¬ 
ate  Groupware — facilitating  the  work  of 
groups  separated  in  space  and  time  (e.g.  the 
Electronic  Meeting  Room). 

Utilities/infrastructure  are  the  third  level  of 
the  hierarchy.  These  include  significant  ex¬ 
isting  capabilities  such  as  moderately  high 
capacity  communications,  data  management 
tools  and  sophisticated  human-machine  in¬ 
terfaces. 

The  fourth  level  of  the  hierarchy  consists  of 


the  applications  being  developed  by  the  acqui¬ 
sition  community.  These  applications  are  char¬ 
acterized  by  the  type  of  tool  (live,  constructive 
or  virtual  simulation)  and  the  functional  disci¬ 
pline  supported  by  the  tool’s  employment  (e.g. 
performance/effectiveness  analyses,  design/ 
engineering).  Efforts  to  integrate  applications 
within  and  across  functional  areas  are  logically 
supported  by  technologies  at  lower  levels  of 
the  notional  hierarchy. 

The  top  level  of  the  hierarchy  is  an  integrated 
acquisition  environment.  This  concept  was 
introduced  in  Chapter  6,  and  is  an  infrastruc¬ 
ture  that  supports  the  information  needs  of 
participants  in  the  acquisition  process  by 
providing  an  integrated  set  of  M&S;  related 
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tools  -  data  base  management,  CAD/CAM, 
and  computer-aided  engineering  (CAE); 
utilities  -  communications  networks  and  data 
repositories;  and  policies  and  procedures  for 
implementation. 

8.5  Management  Actions 

Today’s  managers  must  understand  the  capa¬ 
bilities  and  limitations  of  M&S.  They  must  also 
understand  the  challenges  involved  in  manag¬ 
ing  and  providing  resources  for  the  effort.  As 
a  start,  managers  are  encouraged  to 

•  Recognize  that  M&S  are  powerful  tools 
that  can  offer  significant  opportunities  to 
reduce  the  acquisition  cycle  time  on  a  pro¬ 
gram. 

•  Recognize  that  M&S  are  not  a  pana¬ 
cea  for  all  the  issues  confronting  (and  con¬ 
founding)  DoD’s  weapons  system  acquisi¬ 
tion  process.  Modeling  and  simulation  must 
add  value — not  be  incorporated  just  because 
it  is  “the  in  thing.” 

•  Managers  must  look  down  the  life  cycle 
and  across  the  functional  disciplines  to  iden¬ 
tify  what  specific  functions  M&S  can  per¬ 
form.  Managers  must  also  garner  support 
from  the  various  functional  proponents  for 
the  conduct  or  support  of  specific  tasks  with 
M&S,  using  their  expertise  in  the  identifica¬ 
tion  of  the  tasks. 

•  Gain  user  support  for  the  conduct  of 
specific  user  interface  tasks  with  the  support 
of  M&S.  User  advocacy  is  critical  to  gain  sup¬ 
port  for  a  viable  M&S  effort. 

•  Identify  opportunities  for  achieving 
synergy  through  the  integration  of  models 
and  simulations.  Here  too,  integration  for 
“integration’s  sake”  is  counter-productive. 
Embrace  standards  and  architectures  facili¬ 


tating  interoperability  and  compatibility 
when  they  add  value  to  the  program.  Else,  it 
may  inhibit  management’s  flexibility  in  the 
long  run. 

•  Plan  and  manage  the  M&S  effort  so 
that  it  meshes  completely  with  the  rest  of  the 
program;  becoming  a  part  of  it.  Programs 
whose  acquisition  strategies  are  developed 
to  leverage  the  use  of  models  and  simula¬ 
tions  are  more  likely  to  be  the  most  efficient 
users.  It  is  difficult,  although  not  impossible, 
to  re-structure  an  on-going  program  to  in¬ 
corporate  M&S.  The  biggest  challenge  is  the 
allocation  of  time,  money  and  people.  A 
manager  must  identify  and  acquire  the  re¬ 
sources  required  to  find,  develop  or  modify 
these  tools  as  early  as  possible. 

•  Designate  a  simulation  manager  within 
the  program  office,  to  advise  you  on  all  mat¬ 
ters  pertaining  to  M&S.  As  complex  and  vis¬ 
ible  as  this  area  is  getting,  this  individual 
must  either  have  or  obtain  special  training. 

•  Form  a  Simulation  Working  Group 
(SIMWG),  chaired  by  the  simulation  man¬ 
ager  with  representatives  from  every  element 
of  the  program  team. 

8.6  Some  Application  Best  Practices 

The  purpose  of  this  section  is  to  provide 
some  examples  of  using  M&S  in  the  acqui¬ 
sition  process.  The  weapon  systems  dis¬ 
cussed  in  this  section  are  in  various  phases 
of  acquisition.  However,  the  perceived  ben¬ 
efits  are  impressive  enough  to  give  cause  to 
share  them  with  the  reader. 

8.6.1  Combat  System  Engineering 
and  Analysis  Laboratory  (CSEAL): 

The  CSEAL  is  a  hardware/software-in-the- 
loop  (HW/SWIL)  simulation  facility  at  New- 
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port,  Rhode  Island,  which  allows  for  evalu¬ 
ation  and  rapid  prototyping  of  submarine 
combat  systems.  It  includes  actual  subma¬ 
rine  computer  systems  and  combat  center 
displays  which  allow  for  integration  of  pro¬ 
totype  systems,  along  with  human-in-the- 
loop  response,  decision  making  and  evalua¬ 
tion.  It  has  been  used  in  support  of  the  AN/ 
BSY-2  submarine  combat  system  since  1988. 
This  system  performs  many  functions  includ¬ 
ing  sensor  management  and  sensor  data  pro¬ 
cessing,  sonar  displays,  mission  planning, 
weapons  systems  targeting  and  other  com¬ 
mand  functions  such  as  weather  and  ocean 
condition  predictions. 

The  AN/BSY-2  is  a  major  software  devel¬ 


opment  project  (3.6  million  software  lines 
of  code)  which  has  incorporated  many  sound 
software  development  practices.^  The  use  of 
the  CSEAL  has  been  an  ongoing  part  of  this 
program.  At  the  start  of  the  program,  it  was 
used  in  conjunction  with  Navy  operators  to 
examine  what  functions  the  system  should 
perform  and  how  the  information  should  be 
displayed  to  the  operator.  In  1988  it  was  then 
used  to  develop  design  specifications  from 
the  key  system  operating  functions.  As  parts 
of  the  system  are  developed,  the  CSEAL  has 
been  used  for  independent  verification  and 
validation. 

The  CSEAL  also  has  allowed  data  from  live 
at-sea  exercises  to  be  used  to  stimulate  the 


Combat  System  Engineering  Analysis 
Laboratory  Application  to 
System  Development 


Figure  8-5.  Combat  System  Engineering  Analysis  Laboratory 
Application  to  System  Development 
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AN/BSY-2  system  during  the  development 
process.  Eventually  the  CSEAL  may  support 
design  and  evaluation  of  upgrades  after  the 
system  is  fielded. 

This  application  of  the  CSEAL  demon¬ 
strates  how  the  simulation  environment  can 
support  a  program  across  the  system  life 
cycle.  Figure  8-5  illustrates  further  how  such 
a  simulation  laboratory  can  support  the  span 
of  acquisition  activities;  including  require¬ 
ments  definition,  technology  evaluation,  con¬ 
cept  development  and  system  prototyping.* 

8.6.2  Simulation  support  for  Test  and 
Evaluation  ■  PM  LONGBOW 

The  LONGBOW  is  a  standoff,  helicopter 
launched,  anti-tank  missile.  A  derivative  of 
the  HELLFIRE  missile,  it  uses  the  same  bus, 
but  a  millimeter  wave  seeker  in  place  of 
HELLFIRE’s  laser-guided  seeker.  Re-using 
data  and  models  from  the  HELLFIRE  pro¬ 
gram  and  investing  in  a  hardware-in-loop 
(HWIL)  simulation  facility,  the  LONGBOW 
engineering  and  manufacturing  develop¬ 
ment  (EMD)  flight  test  program  was  re¬ 
duced  (from  50)  to  20  missiles. 

All  test  flights  were  first  simulated  in  the 
HWIL  facility  using  actual  flight  hardware. 
In  several  instances,  potential  flight  failures 
were  found  and  corrected  before  firing  the 
missile  down-range:  resulting  in  time  and 
money  savings. 

When  the  LONGBOW  missile  entered  pro¬ 
duction,  a  Simulation/Test  Acceptance  Facil¬ 
ity  (STAF)  was  built  at  Redstone  Arsenal, 
outside  Huntsville,  Alabama.  The  purpose  of 
the  STAF  was  to  allow  the  LONGBOW  PMO 
to  effectively  execute  a  production  acceptance 
program  and  test  the  seeker  without  firing 
missiles.  Components  such  as  warhead,  mo¬ 
tor  and  battery  are  evaluated  using  the 


HELLFIRE  fly-to-buy  results,  since  these 
components  are  common  to  both  missiles. 

Estimated  annual  savings  from  the  STAF 
is  $7  million,  with  a  payback  period  of 
under  a  year. 

8.6.3  Modeling  and  Simulation  in  Specifi¬ 
cation  Compliance  and  T&E  -  AIM-9X 

The  AIM-9X  missile  is  a  major  improvement 
to  the  existing  family  of  Sidewinder  missiles. 
The  AIM-9X  will  be  used  with  selected  air¬ 
craft  to  detect,  intercept  and  destroy  a  wide 
range  of  high  performance  threat  aircraft. 
This  acquisition  program  is  scheduled  to  be¬ 
gin  Phase  IV/1,  Demonstration  and  Validation 
(DemVal),  in  January  1995.  Extensive  M&S 
will  be  used  throughout  the  life  cycle  of  the 
AIM-9X,  from  source  selection  to  post-pro¬ 
duction  software  support  activities. 

The  M&S  activities  will  be  used  to  reduce 
cost,  reduce  risks  and  ensure  compliance 
with  top  level  system  performance  specifi¬ 
cations.  These  top  level  system  requirements 
include  those  spelled  out  in  the  ORD  in  ad¬ 
dition  to  other  required/desired  perfor¬ 
mance  capabilities,  interface  requirements 
and  “ilities”  requirements. 

Program  cost  reduction  will  be  achieved  by 
using  M&S  to  reduce  the  number  of  missile 
test  firings.  A  philosophy  has  been  adopted 
that  assumes  M&S  will  be  used  to  quantify 
system  performance,  and  that  hardware 
tests,  of  any  kind,  will  be  used  to  generate 
data  for  simulation  validation. 

A  matrix  of  requirements  versus  methods  to 
validate  these  requirements  has  been  devel¬ 
oped.  The  requirements  were  taken  directly 
from  the  requirements  correlation  matrix 
(RCM)  contained  in  the  ORD,  while  meth¬ 
ods  to  validate  the  requirements  were  se- 
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lected  from  proposed  simulations  and  hard¬ 
ware  tests.  A  team  of  people  familiar  with 
simulations,  testing  and  AIM-9X  require¬ 
ments  generated  this  matrix;  identifying  the 
simulations  and  hardware  tests  necessary  to 
validate  each  requirement.  This  process  re¬ 
duced  the  number  of  missile  test  firings  by  a 
factor  of  two  from  the  original  estimate  gen¬ 
erated  using  a  conventional  philosophy  that 
did  not  rely  primarily  on  M&S. 

Simulations  will  be  used  to  ensure  compli¬ 
ance  with  performance  specification  require¬ 
ments.  For  example,  kinematic  performance 
in  the  form  of  launch  acceptability  regions 
(LAR)  is  specified  in  the  system  specifica¬ 
tion,  and  will  be  evaluated  using  a  validated 
six  degrees-of-freedom  (6-DOF)  simulation. 
Program  risk  will  be  reduced  by  ensuring  that 
required  missile  firings  will  have  a  very  high 
probability  of  success  (flight  failures  put  pro¬ 
grams  in  political  jeopardy)  and  by  continu¬ 
ously  monitoring  projected  system  perfor¬ 
mance  to  quickly  identify  issues  that  could 
become  cost  or  schedule  problems. 

Finally,  simulations  will  be  used  to  convert 
hardware  test  results  against  available  assets 
into  equivalent  performance  against  specified 
threats  (i.e.  captive  flight  tests  performance 
against  an  F-18  used  as  a  target  aircraft  must 
be  translated  into  equivalent  performance 
against  the  specified  threat  targets). 

8.6.4  Modeling  and  Simulation  in  Source 
Selection,  New  Start  -  COMANCHE 

Required  by  the  government  to  use  M&S  to 
demonstrate  their  respective  concepts,  con¬ 
tractor  teams  for  the  RAH-66  COMANCHE 
made  extensive  use  of  virtual  prototyping  in 
the  Army’s  newest  aviation  program.  User 
interface  was  established  with  the  contrac¬ 
tor  design  team;  with  Army  test  pilots  flying 
the  virtual  prototype.  Insights  gained  from 


these  flights,  along  with  other  contractual  in¬ 
formation,  permitted  the  PM  to  down-select 
to  a  single  source  for  the  DemVal  phase  of 
the  COMANCHE  program. 

8.6.5  Modeling  and  Simulation  in  Source  Se¬ 
lection,  Non-Developmental  Item  (NDI)  - 
Non-Developmental  Airlift  Aircraft  (NDAA) 

The  Air  Force  has  been  directed  to  evaluate 
the  possible  procurement  of  NDAA  to 
complement  a  reduced  quantity  buy  of  C-17 
strategic  airlifters. 

The  data  used  to  feed  the  Strategic  Airlift 
Force  Mix  Analysis  (SAFMA)  comes  from 
the  Airlift  Loading  Model  (ALM),  and  the 
Mobility  Analysis  Support  System  (MASS). 

The  ALM,  managed  by  Air  Force  Studies 
and  Analysis  (AFSAA),  is  used  to  determine 
the  amount  of  cargo  each  aircraft  can  carry. 
The  model,  in  turn,  depends  upon  data  on 
standard  pallet  dimensions,  vehicle  sizes, 
weights  and  loads;  typically  required  to  meet 
mobilization  requirements.  This  model  is 
provided  to  potential  bidders  to  familiarize 
them  with  tools  to  be  used  in  source  selection. 

The  mobility  analysis  support  system 
(MASS),  managed  by  US  Transportation 
Command,  is  a  time-phased  force  and  de¬ 
ployment  data  (TPFDD)  driven  simulation 
which  considers  the  amount  of  cargo  and 
troops,  as  well  as  factors  such  as  utilization 
rates,  crew  availability  and  airfield  handling 
capacities. 

A  SAFMA  is  being  used  to  provide  infor¬ 
mation  to  the  C-17/NDAA  MS  IIIB  Defense 
Acquisition  Board  (DAB)  pertaining  to 
type(s)  and  number  of  NDAA  to  be  pro¬ 
cured  to  complement  the  C-17  fleet.  This 
cost  and  operational  effectiveness  analysis 
(COEA)-like  analysis  provides  cost  benefit 
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analysis  between  various  possible  mixes  of 
C-17,  NDAA  and  the  current  airlift  fleet. 
The  following  information  is  provided  to 
decision  makers; 

•  Shortfalls  in  Mobility  Requirements 
Study  -  Bottoms-Up  Review  Update  (MRS- 
BURU); 

•  Evaluation  of  force  mixes  for  additional 
aircraft  needed  to  overcome  shortfalls; 

•  Identification  of  options  among  force 
mixes  that  meet  requirements; 

•  Life  cycle  cost  analyses  of  those  options 
found  suitable;  and 

•  Recommendations  on  force  structures 
most  effectively  meeting  requirements. 

Measure  of  effectiveness  (MOE)  are  used 
by  the  SAFMA  to  provide  cost  information 
of  each  aircraft  alternative  mix  and  perfor¬ 
mance;  such  as  throughput  tons  per  day,  clo¬ 
sure  of  combat  units  and  total  force;  which 
could  then  be  assessed  against  higher  level 
measures  of  outcome  (MOO)  such  as  the 
effect(s)  on  a  campaign. 


terns  to  fulfill  the  line-of-sight  forward,  heavy 
(LOS-F-H)  air  defense  mission.  The  RFP 
included  all  requirements  for  the  mission 
and  required  delivery  of  system  specifica¬ 
tions  as  part  of  the  technical  approach;  along 
with  relevant  cost  and  schedule  proposals. 
The  PM’s  approach  was  to  have  the  US 
Army  Missile  Command  (MICOM),  using 
the  expertise  resident  within  their  Research, 
Development  and  Engineering  Center 
(RDEC),  develop  a  simulation  based  upon 
each  of  the  three  design  specifications.  The 
simulations  were  built  to  allow  soldiers  to 
operate  the  system  in  a  synthetic  environ¬ 
ment,  interacting  with  other  virtual  and  con¬ 
structive  models. 

A  crucial  aspect  of  the  PM’s  management 
of  this  effort  was  the  early  involvement  of 
all  team  members  in  a  Simulation  Working 
Group  (SIMWG).  The  SIMWG  included 
representatives  from  the  PMO,  contractor(s), 
user,  model  developer,  tester,  V&V  agency 
and  source  selection  authority. 

Various  aspects  of  the  performance  of  each 
contractor’s  design  in  a  synthetic  environ¬ 
ment  were  included  as  source  selection  cri¬ 
teria,  along  with  cost  and  schedule. 


The  end  result  is  a  cost  effective,  time  con¬ 
servative,  systematic  approach  to  matching 
offerors’  solutions  to  the  user’s  require¬ 
ments.  It  uses  proven  methods  and  validated 
models,  de-emphasizing  the  empirical  ap¬ 
proach,  and  produces  the  COEA  as  an  inte¬ 
gral  part  of  the  overall  study. 

8.6.6  Modeling  and  Simulation  in  Source 
Selection,  Upgrade  -  Bradley-Stinger 

The  Bradley-Stinger  PMO  issued  a  request 
for  proposal  (RFP)  to  three  contractors, 
under  a  limited  comoetition  strategy,  to  ob¬ 
tain  design  specifications  for  candidate  sys- 


8.7  Some  Management  Best  Practices 

8.7.1  Naval  Undersea  Warfare  Center  Cre¬ 
ating  a  Common  Frame  of  Reference 

The  Naval  Undersea  Warfare  Center  has 
implemented  a  series  of  management  practices 
that  might  be  useful  for  the  acquisition  com¬ 
munity  to  consider.  These  include  a  close 
working  relationship  with  the  sponsoring  Pro¬ 
gram  Executive  Office  (PEO)  to  develop  a 
common  frame  of  reference,  CM  practices  and 
archival  of  simulation  runs  and  data. 

In  this  example,  the  Center  and  PEO  staff 


have  worked  together  to  employ  a  common 
frame  of  reference  providing  for  uniform 
application  of  models  and  simulations  across 
all  programs  supporting  that  PEO.  All  sys¬ 
tems  rely  on  the  same  set  of  data  books  which 
contain  the  latest  approved  performance 
characteristics  of  friendly  and  enemy  systems 
and  environmental  data.  Consistent  opera¬ 
tional  tactics  (physical  environment  along 
with  models  validated  with  live  test  data) 
assures  senior  management  that  a  common 
frame  of  reference  is  used  to  evaluate  and 
assess  systems.  Management  does  not  have 
to  repeatedly  question  the  methodologies, 
analysis  approach,  model  validity  or  data 
source.  This  common  framework  also  results 
in  synergy,  or  better  use  of  resources  in  sup¬ 
port  of  all  programs  within  that  warfare  area. 

The  CM  practices  for  computer  simulations 
include  freezing  the  configuration  of  mod¬ 
ules  currently  in  use;  running  test  cases  for 
proposed  changes;  and  finally  documenting 
changes,  developing  manuals  and  notifying 
users  of  updates.  For  the  HWIL  simulators, 
a  formal  CM  plan  is  in  place  that  includes  a 
configuration  control  board.  These  practices 
insure  that  current  model  and  simulation 
configurations  are  properly  validated  and 
documented  for  use. 

An  audit  trail  for  simulations  is  maintained. 
This  includes  archiving  the  actual  model  con¬ 
figuration  and  input  data  so  any  case  can  be 
recreated.  This  is  vital  when  comparison  analy¬ 
sis  using  a  consistent  baseline  is  requested  at  a 
future  date,  or  for  evaluation  of  changes. 

Accomplishing  these  practices  requires  a 
management  commitment.  In  many  cases, 
program  offices  are  focused  on  near-term 
objectives,  but  they  also  should  consider  the 
ongoing  maintenance  which  is  necessary  to 
provide  them,  as  well  as  senior  leadership, 
with  a  capability  at  a  future  date. 


8.7.2  Modeling  and  Simulation  Manage¬ 
ment  -  BAT 

Originally  “Brilliant  Anti-Tank,”  the  BAT 
sub-munition  was  being  developed  to  kill 
second  and  third  echelon  enemy  moving  tank 
formations.  The  BAT’s  system  effectiveness 
was  defined  using  672  different  conditions. 
A  test  program  involving  statistically  valid 
information  for  each  condition  would  have 
been  unaffordable.  Faced  with  the  task  of 
designing  a  complex  and  prohibitively  expen¬ 
sive  test  program,  the  PMO  received  DAB 
approval  to  evaluate  the  BAT  performance 
and  effectiveness  using  simulations  subjected 
to  rigorous  verification,  validation  and  ac¬ 
creditation  (VV&A). 

The  BAT  program  uses  a  family  of  simula¬ 
tions  that  collectively  contribute  to  system 
effectiveness  prediction. 

The  BAT  system  validation  process,  shown 
in  Figure  8-6,  is  iterative;  using  test  data  to 
validate  the  simulation.  Furthermore,  the 
BAT  PMO  developed  its  system  simulation 
independent  from  the  prime  contractor,  but 
from  the  same  specifications.  Each  model  is 
used  as  a  cross-check  on  the  other. 

The  BAT  test  methodology,  shown  in  Fig¬ 
ure  8-7,  uses  the  model-test-model  concept 
to  ensure  that  the  system  model  is  kept  in 
sync  with  the  tactical  design.  Predictions 
from  pre-test  runs  of  the  model  are  com¬ 
pared  with  post-test  analyses  to  ensure  that 
any  differences  can  be  explained,  or  form  the 
basis  for  a  modification. 

The  BAT  PMO  formed  a  SIMWG  consist¬ 
ing  of  a  wide  constituency  and  a  team  ap¬ 
proach  to  validation  and  accreditation.  Ac¬ 
ceptance  of  simulation  results  in  support  of 
operational  testing  is  simplified  through 
shared  responsibility  of  the  operational  test 
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accreditation  process  by  the  BAT  PMO  and 
the  sponsor  (Army’s  Operational  Test  and 
Evaluation  Command). 

The  BAT  SIMWG  implemented  a  solid  CM 
approach  involving  configuration  baselining, 
change  control,  evaluation  and  processing. 
Release  of  simulations  is  controlled  through 
execution  of  a  Memorandum  of  Understand¬ 
ing  (MOU)  with  each  user;  as  well  as  main¬ 
tenance  of  a  data  base  of  users,  what  they 
have  (documentation,  software,  sample 
cases)  and  the  simulation  version. 

The  BAT  program  is  recognized  as  a  case 
study  in  the  use  of  teaming  to  accomplish 
effective  use  and  control  of  performance  and 
effectiveness  M&S. 


8.7.3  Simulation  Management  - 
TOMAHAWK  Cruise  Missile 

The  TOMAHAWK  cruise  missile  program 
has  had  a  simulation  management  board  in 
place  for  approximately  ten  years.  This  simu¬ 
lation  management  provides  many  functions 
which  include; 

•  Maintaining  a  simulation  catalog  con¬ 
taining  a  description  of  each  simulation, 
technical  status,  limitations,  functions  and 
current  utilization.  Entry  into  the  catalog  is 
part  of  the  simulation  certification  process 
and  only  those  simulations  in  the  catalog  may 
be  used  to  define  performance  capability  and 
system  effectiveness  estimates. 
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BAT  Test  Methodology 


MODEL -TEST  MODEL 


Figure  8-7.  BAT  Test  Methodology 


•  Integrating  individual  simulation  status 
and  plans  within  TOMAHAWK  program 
plans. 

•  Maintaining  a  set  of  authorized  refer¬ 
ence  simulations  which  serve  as  performance 
baselines  for  the  missile  and  for  evaluation 
of  proposed  changes.  Reference  check  cases 
are  defined  with  inputs,  initial  conditions, 
assumptions  and  expected  outputs;  typically 
validated  by  test  data. 

•  A  method  of  documenting/disclosing 


the  pedigree  (certification)  status  of  a  simu¬ 
lation:  a  certificate  which  is  approved  by  the 
chairman  of  the  simulation  management 
board.  This  certificate,  shown  in  Figure  8-8, 
is  used  when  publishing  or  presenting  per¬ 
formance  analysis  results. 

The  value  of  this  management  process  is 
coordination  of  simulation  plans  across  the 
entire  program,  providing  standard  refer¬ 
ence  simulations  and  check  cases  and  cen¬ 
tral  management  of  certified  TOMAHAWK 
models  and  data  bases  for  use;  eliminating 
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TOMAHAWK  Simulation  Management  Board  Certificate 


To:  (Responsible  Organization) _ 

From:  TOMAHAWK  Simulation  Management  Board 

Subject:  (Full/Provisional/Limited)  Certification  of  TOMAHAWK _ (Simulation  Name) 

Certified  to  Represent  -  for  use  in: _ 

(Define  domain  of  certification  appiicabifity) 

Responsible  Organization: _ 


This  (Full/Provisional)  Certification  is  for; 

The  following  materials  support  certification; 

Version  Dated 

Certification  Plan 

Users  Manual 

_ Certification  Report 

_ Benchmark 

Provisional  Pending; 

_ Configuration 

Checkcases 

Control  Plan 

_ Reference 

Based  on  the  following  major  activities; 

Checkcases 

_ Validation  against  Flight  Test  Results 

__Current  Catalog 

_ Source 

Validation  against 

Entry 

Code/Listing 

Other 

Previous  Certification;  Version  Dated 

Certification  Recommended:  Certification  Approved: 

SCP  Chairman; _  _ 

Chairman.  Simulation  Management  Board,  Date 

SCP  Co-Chair; _ 


Figure  8-8.  TOMAHAWK  Simulation  Management  Board  Certificate 


redundancy  of  multiple  simulation  develop¬ 
ment  efforts  and  potential  errors.^ 

8.8  Summary 

The  instances  discussed  in  the  preceding  sec¬ 
tions  are  only  representative  of  the  many 
successful  uses  of  M&S  in  systems  acquisi¬ 
tion.  The  cases  mentioned,  however,  rein¬ 


force  a  significant  aspect  of  the  use  of  M&S 
in  acquisition.  Careful  early  planning  and 
investment  for  the  use  of  M&S  will  pay  divi¬ 
dends  through  cost  avoidance.  While  cost 
avoidance  is  not  as  easy  to  quantify  or 
project,  as  cost  savings,  the  results  for  an  in¬ 
stitution  facing  a  downward  trend  in  bud¬ 
gets,  are  just  as  tangible. 
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EPILOGUE 


The  use  of  modeling  and  simulation  (M&S)  in  defense  acquisition  is  certain  to  grow  dra¬ 
matically  in  coming  years.  Advances  in  simulated  battlefield  environments  will  support 
models  possessing  the  resolution  and  fidelity  to  address  the  complex  issues  at  hand.  The 
extension  of  innovative  implementations  of  advanced  concepts,  such  as  Distributed  Inter¬ 
active  Simulation  (DIS),  are  creating  exciting,  cost  effective  means  for  the  conduct  of 
wargaming,  training,  and  requirements  and  concept  development.  The  use  of  M&S  tools  is 
expanding  within  each  functional  domain  of  weapon  systems  acquisition.  Integration  of 
tools  used  among  the  functional  domains  is  beginning  as  a  natural  follow-on. 

The  primary  intent  of  this  document  is  stated  in  the  title.  However,  the  authors  feel  com¬ 
pelled  to  include  a  vision  of  how  things  may  be  in  the  not-too-distant  future,  as  well  as  to 
reiterate  some  advice  to  Program  Managers  (PMs)  and  policy  makers. 

Economic,  political,  national,  and  international  forces  are  responsible  for  much  of  the  re¬ 
cent  reorganization  and  process  re-engineering  within  the  Department  of  Defense  (DoD). 
During  the  conduct  of  this  research,  we  witnessed  new  ideas  germinating  and  products 
taking  shape  in  response  to  these  changes.  These  ideas  are,  largely,  the  result  of  a  recogni¬ 
tion  that  DoD  must  use  its  resources  with  greater  efficiency.  An  inevitable  result  of  these 
innovations  will  be  a  better,  more  focused  and  timely  allocation  of  resources. 

A  logical  next  step  is  an  intense  focus  on  mission  areas  and  their  requirements  from  a  truly 
joint  perspective.  For  example,  if  all  services  participate  in  the  counter-air  mission  area, 
why  not  approach  the  Mission  Area  Assessment  (MAA)  and  Mission  Needs  Analysis  (MNA) 
from  an  all-service  contribution  standpoint?  The  potential  for  M&S  to  play  a  major  role  in 
contributing  to  such  analyses,  is  significant. 

Individual  PMs  must  understand  what  M&S  tools  can  do  for  their  programs,  as  well  as 
recognize  their  limitations.  They  (M&S  tools)  offer  significant  benefits,  but  are  not  a  pana¬ 
cea.  The  PMs  must  recognize  that  the  management  of  M&S  efforts  require  resource  in¬ 
vestments,  and  that  the  resulting  benefits  are  usually  in  cost  avoidance  rather  than  cost  sav¬ 
ings.  Each  PM  must  also  recognize  that  any  program  will  normally  require  the  support  of 
several  models  and  simulations,  and  that  additional  benefits  may  accrue  from  the  integra¬ 
tion  of  some  of  these  tools. 

Policy  makers,  on  the  other  hand,  must  recognize  the  PM’s  lot.  Efforts  to  institutionalize 
the  use  of  M&S  in  weapon  systems  acquisition  must  be  based  on  an  assessment  of  the  value 
added.  Guidance  should,  and  must,  be  provided.  However,  it  must  be  judicious  in  its  man¬ 
date.  PMs  must  be  provided  the  latitude  to  do  what  is  smart  for  their  individual  programs. 

Policy  that  adds  baggage  without  creating  value  will  stifle  the  very  innovation  so  crucial  to 
achieving  the  synergy  that  the  application  of  M&S  tools  is  capable  of  providing. 


APPENDIX  A 

DoD  SOURCES  OF  INFORMATION  FOR  MODELING 
AND  SIMULATION  IN  WEAPON  SYSTEMS  ACQUISITION 


This  appendix  contains  a  listing  of  points  of  contact  for  modeling  and  simulation  within  the 
Office  of  the  Secretary  of  Defense  (OSD),  Department  of  Defense  (DoD)  Agencies  and 
the  Joint  Staff. 


Defense  Modeling  and  Simulation  Office  (DMSO) 

Phone:(703)998-0660 

Advanced  Research  Projects  Agency  (ARPA) 

Advanced  Systems  Technology  Office  (ASTO) 
Phone:(703)696-2309 
DSN:  226-2309 

Ballistic  Missile  Defense  Office  (BMDO) 

Modeling  and  Simulation  Directorate  (AQM) 
Phone:(703)693-1594 
DSN:  223-1594 

Defense  Intelligence  Agency  (DIA) 
Acquisition/Plans  Intelligence  Support  Division  (PAN-2) 
Phone:(202)373-3101 
DSN:  243-3101 

Defense  Information  Systems  Agency  (DISA) 
Center  for  Standards 

C3I  Support  Division  (TBC) 
Phone:(703)487-3538 
DSN:  364-3538 

Information  Processing  Standards  Division  (TBE) 
Phone:(703)487-3552 
DSN:  364-3552 

Defense  Logistics  Agency  (DLA) 

Warfighting  and  Integration  Division  (CAILW) 
Phone:(703)274-6715 
DSN:  284-6715 
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Defense  Mapping  Agency  (DMA) 

Plans  and  Requirements  Directorate 
Advanced  Weapon  &  Systems  Division  (PRW) 
Phone:(703)285-9326 
DSN:  356-9326 


Defense  Nuclear  Agency  (DNA) 

Operational  Support(OPNA) 
Phone:(703)325-7177 
DSN:  221-7177 

National  Defense  University  (NDU) 

War  Gaming  and  Simulation  Center  (NDU-NSS-WGSC) 
Phone:(202)475-1251 
DSN:  335-1251 


THE  JOINT  STAFF 

Director  for  Logistics  (J4) 

Mobility  Division  (J4/MOB) 

Phone:(703)697-6110 
DSN:  227-6110 

Director  for  Command,  Control,  Communication,  and  Computer  Systems  (J6) 

Resource,  Planning,  and  Evaluation  Division  (J6E) 
Phone:(703)697-8590 
DSN:  227-8590 

C4  Architecture  and  Integration  Division  (J6I) 
Phone:(703)614-8787 
DSN:  224-8787 

Director  for  Operational  Plans  and  Interoperability  (J7) 

Joint  Simulation  Division  (J7/JSID) 

Policy 

Phone:(703)695-3047 
DSN:  225-3047 

Joint  Test! Evaluation 
Phone:(703)693-3418 
DSN:  223-3418 


A-2 


Joint  Warfighting  Center  (JFWC-CC) 
Phone:(904)884-7720 
DSN:  579-7720 


Director  for  Force  Structure,  Resources,  and  Assessment  (J8) 
Automated  Systems  Division  (J8/ASD) 

Applications 
Phone:(703)697-7824 
DSN:  227-7824 

Data  Bases 
Phone:(703)697-8899 
DSN:  227-8899 

mW  Configuration 
Phone:(703)693-4614 
DSN:  223-4614 


Resources  and  Joint  Planning  Division  (J8/RJPD) 


Contracts 

Phone:(703)614-7881 
DSN:  224-7881 


Contracted  Advisory  and  Assistance  Service  (CAAS) 
Phone:(703)693-4608 
DSN:  223-4608 


Modem  Aids  to  Program  Planning  (MAPP) 
Phone:(703)695-1763 
DSN:  225-1763 


System  Programs  Evaluation  Division  (J8/SPED) 
Phone:(703)697-6299 
DSN:  227-6299 


CATALOGS; 

Catalog  of  Wargaming  and  Military  Simulation  Models.  (1992).  (12th  ed.)  Washington,  DC: 
Force  Structure,  Resource,  and  Assessment  Directorate,  Technical  Support  and 
Operations  Division  (J-8/TSOD),  The  Joint  Staff. 
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APPENDIX  B 

DEPARTMENT  OF  THE  ARMY  SOURCES  OF  INFORMATION 
FOR  MODELING  AND  SIMULATION 
IN  WEAPON  SYSTEMS  ACQUISITION 


This  appendix  is  organized  primarily  to  assist  Army  acquisition  managers  in  their  search 
for  information  regarding  modeling  and  simulation  (M&S)  activities  of  various  Army  orga¬ 
nizations.  It  provides  phone  numbers,  office  symbols  and  addresses  to  assist  users  in  mak¬ 
ing  initial  contact.  This  appendix  also  provides  information  on  MOdel  and  Simulations: 
Army  Integrated  Catalog  (MOSAIC).  Figure  B-1  depicts  the  relationship  of  organizations 
active  in  the  Army’s  development,  management  and  use  of  M&S  in  acquisition. 


Figure  B-1 .  Army  Organizations  Active  in  M&S  for  Acquisition 


POLICY  AND  GUIDANCE 

Army  Model  and  Simulation  Management  Office  (AMSMO):  The  AMSMO  promulgates 
Army  M&S  policy;  publishes  guidance  and  administers  the  Army  Model  Improvement 
Program  (AMIP)  and  Simulation  Technology  Program  (SIMTECH);  develops  and  pub¬ 
lishes  the  Army  Model  and  Simulation  Master  Plan  and  implementing  procedures  such  as 
the  verification,  validation  and  accreditation  (VV&A)  of  Army  models  and  simulations.(The 
AMSMO  will  be  the  focal  point  for  establishing  and  maintaining  the  media  by  which  to 
identify  and  share  information  on  agencies,  organizations  and  activities  in  the  Army  com- 
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munity  who  use  or  develop  M&S.)  The  point  of  contact  (POC)  is; 

U  S.  AMSMO 

1725  Jefferson  Davis  Highway,  Suite  808 
Arlington,  VA  22202 
Phone;(703)607-3375 

POCs  FOR  ANALYSIS  MODELS  (BY  FORCE  LEVEL) 

System  Performance  Models,  Item  level:  Focusing  on  a  single  weapon  system  or  piece  of 
equipment.  Examples  include;  Air  Defense  Air-to-Ground  Engagement  (ADAGE),  Artil¬ 
lery  Force  Simulation  Model  (AFSM),  Simplified  Artillery,  Reliability  Growth  model  (e.g. 
SESAME),  Projectile  Effectiveness  Model  (e.g.  ARTQUIK),  TANKWARS,  NATO  Refer¬ 
ence  Mobility  Model.  The  POC  is; 

U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 

Special  Studies  and  Activities  Office 
ATTN;AMXSY-DA 

Aberdeen  Proving  Ground,  MD  21005-5071 
Phone;(410)278-6576 
DSN;  298-6576 

Brigade  and  task  force  level,  and  below:  Focusing  on  combined  arms  forces  and  single 
functional  elements;  they  are  represented  as  an  integral  part  of  combined  arms  and  ser¬ 
vices  activities.  Examples  include;  Combined  Arms  and  Support  Task  Force  Evaluation 
(CASTFOREM),  JANUS(T),  ELANT,  American  Canadian  Australian  British  Urban  Game 
(ACABUG).  The  POC  is: 

Commander,  U.S.  Army  TRADOC  Analysis  Command  (TRAC) 
Brigade/Battalion  Modeling  and  Analysis  Support  Center 
ATTN:ATRC-WE 

White  Sands  Missile  Range,  NM  88002-5502 
Phone:(505)678-1012 
DSN;  258-1012 

Corps  and  Division  Level:  Focusing  on  single  and  multi-division  levels  of  operation  with  or 
without  a  supervising  corps  headquarters.  Examples  include:  EAGLE,  Corps  Battle  Ana¬ 
lyzer  (CORBAN),  Vector-in-Commander  (VIC),  JIFFY.  The  POC  is: 

Commander,  U.S.  Army  TRADOC  Analysis  Command  (TRAC) 
Corps/Division  Modeling  and  Analysis  Operations  Analysis  Directorate  Support  Center 

ATTN:ATRC-OAC 
Ft.  Leavenworth,  KS  66027 
Phone:(913)684-2276 
DSN;  552-2276 
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Theater  level  and  above:  Focusing  on  all  force  levels  at  echelons  above  corps;  includes 
multi-corps,  regional  and  global  models  and  simulations.  Examples  include:  Force  Evalua¬ 
tion  Model  (FORCEM),  Concepts  Evaluation  Model  V  (CEM  V),  Force  Analysis  Simula¬ 
tion  of  Theater  Administrative  and  Logistic  Support  (FASTALS),  Transportation  Model 
(TRANSMO).  The  POC  is: 

Commander,  U.S.  Army  Concepts  Analysis  Agency  (CAA) 

Research  and  Analysis  Support  Directorate 
ATTN:CSCA-RS 
8120  Woodmont  Ave 
Bethesda,  MD  20814 
Phone;(301)295-1692 
DSN;  295-1692 


POCs  BY  M&S  FUNCTIONAL  APPLICATION 

Engineering  Models:  Information  focusing  on  models  and  simulations  which  augment  de¬ 
sign,  engineering  and  testing  in  various  stages  of  the  materiel  acquisition  process.  Models 
and  simulations  are  used  in  investigating  mechanical,  electrical  and  physical  phenomena 
associated  with  the  functioning  of  an  item  or  system.  Examples  include  hardware-in-the- 
loop  (HWIL),  six  degree-of-freedom  (6-DOF),  physics-of-failure  simulations,  computer- 
aided  design  (CAD),  computer-aided  manufacturing  (CAM)  and  phenomenology  models. 
The  POCs  at  the  various  Research,  Development  and  Engineering  Centers  (RDEC)  are: 

Commander,  U.S.  Army  Communications-Electronics  Command  (CECOM) 
Research,  Development  and  Engineering  Center  (CRDEC) 
ATTN:AMSEL-RD-ST-CE-M 
Ft.  Monmouth,  NJ  07703-5203 
Phone:(908)544-4708 
DSN:  992-4708 

Commander,  U.S.  Army  Close  Combat  Armament  Center 
Light  Armament  Division 
ATTN:SMCAR-CCL-E 
Picatinny  Arsenal,  NJ  07806-5000 
Phone:(201)724-6054 
DSN:  880-6054 

Commander,  U.S.  Army  Fire  Support  Armament  Center 
Fire  Control  Division 
ATTN:SMCAR-FSF-BD 
Picatinny  Arsenal,  NJ  07806-5000 
Phone:(201)724-7920 
DSN:  880-6054 
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Commander,  U.S.  Army  Missile  Command  (MICOM) 
ATTN:AMSMI-RD-SS 
Redstone  Arsenal,  AL  35898-5252 
Phone;(205)876-4271 
DSN:  746-4271 

Commander,  U.S.  Army  Tank-Automotive  Command  (TACOM) 
ATTN:AMSTA-RYA 
Warren,  MI  48397-5000 
Phone:(313)574-8633 
DSN:  786-8683 

Commander,  U.S.  Army  Aviation  and  Troop  Command  (ATCOM) 

ATTN:AMSAT-R-B 
4300  Goodfellow  Blvd. 

St.  Louis,  MO  36120-1798 
Phone:(314)263-1333 
DSN:  693-1333 

Commander,  U.S.  Army  Edgewood  RD«feE  Center 
ATTN:SCBRD-RTM 

Aberdeen  Proving  Ground,  MD  21010-5423 
Phone:(410)671-1774 
DSN:  584-1774 

Commander,  U.S.  Army  Natick  RD&E  Center 
ATTN:SATNC-AAM 
Kansas  Street 
Natick,  MA  01760-5015 
Phone:(508)651-4881 
DSN:  256-4881 

Director,  U.S.  Army  Research  Laboratory  (ARL) 

2800  Powder  Mill  Road 
Adelphi,  MD  20783-1145 
Phone:(301)394-4650 
DSN:  290-4650 

Cost  Models:  The  US  Army  Cost  and  Economic  Analysis  Center  (CEAC)  develops  and 
screens  models  that  pertain  to  financial  management  (e.g.  cost  analysis);  designs  and  imple¬ 
ments  the  cost  methodology  for  program  office  estimates  and  component  cost  analysis 
that  shape  the  Army  cost  position.  The  CEAC  develops  data  bases,  cost  models  and  cost 
estimating  relationships  for  major  materiel  systems;  tracks  operating  and  support  costs; 
captures  demand  data  and  scales  it  to  activity  or  use;  uses  M&S  to  cost  real  time  sustainment 
cost  during  the  battlefield  simulations;  develops  and  screens  M&S  pertaining  to  manpower 
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costs;  and  tracks  the  personnel  costs  to  operate  the  force.  The  POC  is: 

Director,  U.S.  Army  Cost  and  Economic  Analysis  Center  (CEAC) 
ATTN:SFFM-CA-CR 
5611  Columbia  Pike,  Room  436 
Falls  Church,  VA  22041-5050 
Phone:(703)756-8732 
DSN:  289-8732 

Logistics  Models:  The  U.S.  Army  Logistics  Support  Agency  (LOGSA)  develops  and  man¬ 
ages  techniques/models  for  performing  logistics  support  analysis  (LSA)  and  logistics  sup¬ 
port  analysis  reports  (LSAR);  is  responsible  for  policy,  techniques/  models  and  data  ele¬ 
ments  for  level  of  repair  analysis  (LORA);  is  proponent,  developer  and  configuration  man¬ 
ager  of  three  of  the  currently  designated  Army  standard  LORA  techniques/models;  per¬ 
forms  LORA  on  designated  weapon  systems;  and  provides  consultation  and  guidance  on 
performing  LORA  for  all  Army  weapon  systems.  The  POCs  are: 

Executive  Director,  AMC  LOGSA 
ATTN:  (See  Below) 

Redstone  Arsenal,  AL  35878-7466 
Phone  Hotline:  1-800-553-0764/0769 

For  LORA:  ATTN:AMXLS-AL 
FOR  LSA/LSAR:  A7TN;AMXLS-ALD 

Manpower  Models:  The  Office  of  the  Deputy  Chief  of  Staff  for  Personnel  (DCSPER),  Direc¬ 
tor  for  Manpower  and  Personnel  Integration  (MANPRINT)  has  oversight  for  development  of 
M&S  for  soldier  survivability  and  manpower,  personnel  and  training  integration.  The  POC  is: 

HQDA,  ODCSPER,  Director  for  MANPRINT 
ATTN:  DAPE-MR 
The  Pentagon 

Washington,  DC  20310-0300 
Phone:(703)697-9213 
DSN:  227-9213 

Manufacturing  Models:  Since  these  models  are  usually  weapon  system,  material  and  tech¬ 
nology  specific,  a  central  POC  is: 

Commander,  U.S.  Army  Materiel  Command 
Manufacturing  Science  and  Technology  Office 
ATTN:AMCRD-IEM 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-0001 
Phone:(703)274-9437 
DSN:  284-9437 
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TESTING: 


Developmental  Testing:  The  various  RDECs  possess  inherent  capabilities  to  conduct  tech- 
I  nical  testing  at  various  stages  of  the  acquisition  process.  To  support  formal  developmental 

testing,  the  U.S.  Army  Test  and  Evaluation  Command  (TECOM)  develops  and  supports 
M&S  for  vibration  and  environmental  testing;  uses  M&S  for  meteorology  studies  for  test 
planning  and  to  drive  simulation  targets  in  moving  target  simulators;  and  is  developing  a 
virtual  test  range  in  support  of  virtual  prototyping  and  other  simulation  exercises.  The 
POC  is: 


Commander,  TECOM,  Technology  Development  Division 
ATTN:AMSTE-CT-T 

Aberdeen  Proving  Ground,  MD  21005-5055 
Phone:(410)278-1479 
DSN:  298-1479 

Operational  Testing:  The  U.S.  Army  Operational  Test  and  Evaluation  Command  (OPTEC) 
is  the  Army’s  lead  agent  for  operational  test  and  evaluation;  using  M&S  throughout  the 
testing  process,  OPTEC  applies  cost  effective  techniques  to  situations  of  limited  testability; 
and  incorporates  the  notion  of  distributed  interactive  simulation  (DIS)  through  the  use  of 
constructive  and  virtual  simulations  in  the  live  simulations  (operational  tests)  that  it  con¬ 
ducts.  The  POC  is: 

Commander,  U.S.  Army  OPTEC,  Office  of  Policy  and  Methodology 

ATTN:CSTE-MP 
Park  Center  IV,  4501  Ford  Avenue 
Alexandria,  VA  22302-1458 
Phone:(703)756-1685/1688 
DSN:  289-1685/1688 

IVaining:  The  Simulation,  Training  and  Instrumentation  Command  (STRICOM)  manages 
the  development  and  acquisition  of  simulations  to  support  training,  exercises,  and  military 
operations.  The  various  Program  Managers  are:  PM  -Training  Devices  (PM-TRADE);  PM 
-  Instrumentation,  Targets  and  Threat  Simulators  (PM-ITTS);  PM  -  Combined  Arms  Tac¬ 
tical  Trainer  (PM-CATT);  and  PM  -  Distributed  Interactive  Simulation  (PM-DIS).  Central 
POC  is: 


Commander,  U.S.  Army  STRICOM,  Director  for  Management 

ATTN:AMSTI-M 
12350  Research  Parkway 
Orlando,  FL  32826-3276 
Phone:(407)380-8234 
DSN:  960-8234 
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M&S  DATA  SOURCES 

Weapon  Systems  Performance  Data:  Item  level  weapon  systems  performance  data  for  U.S. 
and  threat  systems,  and  characteristics  data  for  U.S.  systems.  Data  focusing  on  the  lowest 
level  system,  such  as  a  gun  with  its  crew  or  a  tank  with  its  crew  (includes  reliability  and 
supportability).  The  POC  is: 

U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 

Special  Studies  and  Activities  Office 
ATTN:AMXSY-DA 

Aberdeen  Proving  Ground,  MD  21005-5071 
Phone:(410)278-6576 
DSN:  298-6576 

Threat  Systems  Performance  Data:  Item  level  weapon  systems  operational  and  character¬ 
istic  data  for  threat  systems  focusing  on  lowest  level  threat  systems,  such  as  an  air  defense 
gun  or  tank.  The  POC  is: 

HQDA,  Office,  Deputy  Chief  of  Staff  for  Intelligence  (ODCSINT) 

ATTN:DAM1-FIT 
The  Pentagon 

Washington,  DC  20310-1088 
Phone:(703)614-8121 
DSN:  224-8121 


Weather  Data:  The  POC  is: 


Director,  U.S.  Army  Research  Laboratory  (ARL) 
2800  Powder  Mill  Road 
Adelphi,  MD  20783-1145 
Phone:(301)394-4650 
DSN:  290-4650 


Terrain  Data:  The  POC  is: 


Director,  U.S.  Army  Topographic  Engineering  Center  (TEC) 
7701  Telegraph  Road 
Alexandria,  VA  22310-3864 
Phone:(703)355-3176 
DSN:  345-3176 
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Smoke  Data:  The  POC  is: 


Director,  U.S.  Army  Research  Laboratory  (ARL) 
2800  Powder  Mill  Road 
Adelphi,  MD  20783-1145 
Phone:(301)394-4650 
DSN:  290-4650 


OTHER  ORGANIZATIONS  WITH  SPECIFIC 
M&S  RESPONSIBILITIES  AND  CAPABILITIES 

U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA):  The  AMSAA  uses  M&S  to  pro¬ 
vide  U.S.  and  threat  systems  performance  data  for  use  in  cost  and  operational  etfectiveness 
analyses  (COEA),  in  Army  studies,  and  in  support  of  the  acquisition  of  systems;  helps 
accredit  and  provides  certified  systems  performance  data  characteristics  and  data  to  the 
Army  community;  provides  VV&A  support  to  U.S.  Army  Materiel  Command  (AMC)  and 
other  agencies;  promulgates  VV&A  and  data  certification  policy  throughout  AMC;  and,  in 
conjunction  with  TRADOC,  standardizes  data  and  algorithms  within  the  Army  commu¬ 
nity.  The  AMSAA  maintains  configuration  control  on  M&S  for  item  level  performance, 
one-on-one  system  performance,  few-on-few  and  many-on-many  combat  and  large  war 
gaming  simulation.  The  POC  is: 

U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 

Special  Studies  and  Activities  Office 
ATTN:AMXSY-DA 

Aberdeen  Proving  Ground,  MD  21005-5071 
Phone:(410)278-6576 
DSN;  298-6576 

U.S.  Army  Industrial  Operations  Command  (IOC):  Formerly  the  U.S.  Army  Armament, 
Munitions  and  Chemical  Command.  The  IOC’s  Cost  and  Systems  Analysis  Directorate 
provides  and  uses  M&S  to  assess  manufacturing  and  flexible  computer  integrated  manu¬ 
facturing  and  repair  operations  at  the  macro  and  micro  level  within  the  arsenals,  the  depots 
and  the  ammunition  plants.  The  POC  is: 

Commander,  Industrial  Operations  Command 
ATTN:AMSMC-AN 
Rock  Island,  IL  61299-6000 
Phone:(309)782-5262 
DSN;  793-5262 

National  Simulation  Center  (NSC):  The  TRADOC  proponent  for  all  battle  command  train¬ 
ing  simulations.  The  NSC  sponsors,  designs,  maintains  and  provides  configuration  man¬ 
agement  (CM)  of  all  Army  battle  command  training  M&S;  and  coordinates,  manages,  and 
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executes  training  simulation  V&V.  The  POC  is; 

Commander,  Combined  Arms  Center 
ATTN  ;ATZL -NSC 
Ft.  Leavenworth,  KS  66027-7305 
Phone:(913)684-8101 
DSN:  552-8101 

TRADOC  Battle  Labs:  The  Battle  Labs  use  models  and  simulations,  as  well  as  other  tools, 
to  improve  battlefield  capabilities  by  deriving  insights  across  doctrine,  training,  leader  de¬ 
velopment,  organization  design  and  materiel  (DTLOM).  Centra!  POC  is; 

Commander,  U  S.  Army  TRADOC 
ATTN;ATCD-B 
Ft.  Monroe,  VA  23651-5000 
Phone:(804)727-4283/4284 
DSN;  680-4283/4284 

U.S.  Army  Space  and  Strategic  Defense  Command  (USASSDC):  The  USASSDC  applies 
M&S  in  conducting  research  and  development  (R&D)  of  missile  defense  simulations.  The 
POC  is: 

Commander,  Space  and  Strategic  Defense  Command 
ATTN:CSSD-CR 
PO.  Box  1500 
Huntsville,  AL  35807-3801 
Phone;(205)955-1354 
DSN:  654-1354 

U.S.  Army  War  College  (AWC):  The  AWC’s  Center  for  Strategic  Leadership  uses  M&S  in 
support  of  students  and  curriculum.  The  Center  uses  M&S  for  war  gaming,  simulation 
support,  and  studies;  analysis  for  other  Army  and  DoD  commands  and  agencies  at  the 
operational  and  strategic  levels  of  war;  and  provides  limited  development  of  M&S  educa¬ 
tion  decision  tools.  The  POC  is: 

Commandant,  U.S.  Army  War  College 
ATTN:AWC-AW 
Carlisle  Barracks,  PA  17013-5050 
Phone:(717)245-4281 
DSN;  242-4?81 
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THE  MODELS  AND  SIMULATIONS:  ARMY  INTEGRATED  CATALOG  (MOSAIC) 

The  MOSAIC  provides  potential  model  developers  and  users  the  opportunity  to  peruse 
the  array  of  existing  M&S  and  query  the  hypertext  system  for  all  information  of  interest  to 
them  in  their  proposed  application.  The  official  registry  of  M&S,  MOSAIC  contains  all 
active  and  developmental  M&S. 

In  AR  5-11,  Army  M&S  proponents  are  directed  to: 

•  Enter  information  about  their  M&S  into  the  catalog 

•  Keep  that  information  current  by  providing  updates  at  least  every  two  years 
To  become  and  remain  a  valid  entry  in  MOSAIC,  a  model  or  simulation  must: 

•  Be  computerized 
The  POC  for  MOSAIC  is: 

Chief,  Army  Model  and  Simulation  Management  Office  (AMSMO) 
ATTN:SFUS-MIS  (MOSAIC  System  Administrator) 

1725  Jefferson  Davis  Highway,  Suite  808 
Arlington,  VA  22202 
Phone:(703)607-3383 
DSN:  327-3383 


ARMY  POLICY  AND  PROCEDURE  REFERENCES 

Array  Regulation  5-11,  Army  Model  and  Simulation  Management  Program  (AMSMP):  Pre¬ 
scribes  policy  for  Army  M&S  management  and  formalizes  the  Army’s  program  for  man- 
agemeiit  of  models  and  simulations. 

Department  of  The  Army  Pamphlet  5-11,  Verification,  Validation,  and  Accreditation  of 
Army  Models  and  Simulations:  Provides  procedures  to  assist  model  developers,  propo¬ 
nents  and  sponsors  to  conform  to  the  policies  in  AR  5-11. 

Army  Model  and  Simulation  Master  Plan:  Promotes  the  adoption  of  standards  and  com¬ 
mon  tools  and  processes  in  building  and  populating  models  and  simulations,  for  use  in  all 
applications  throughout  the  Army. 

OPTEC  Handbook  73-21;  Written  primarily  for  OPTEC’s  evaluators,  test  officers  and  ana¬ 
lysts.  It  assists  OPTEC  personnel  in  coordinating  more  proficiently  with  external  parties 
with  respect  to  the  use  of  M&S  in  support  of  OT&E.  Likewise,  it  allows  managers  to  be 
pro-active  in  their  M&S  planning  efforts. 
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APPENDIX  C 

DEPARTMENT  OF  THE  NAVY  SOURCES 
OF  INFORMATION  FOR  MODELING  AND  SIMULATION 
IN  WEAPON  SYSTEMS  ACQUISITION 


This  appendix  provides  information  on  various  organizations  within  the  Department  of  the 
Navy  (DON)  that  are  involved  with  modeling  and  simulation  (M&S)  activities.  It  provides 
phone  number,  office  codes  and  addresses  to  assist  users  in  making  initial  contact. 


Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

The  ASN(RDA)  has  overall  responsibility  for  Navy  acquisition.  It  provides  guidance  on 
development  and  use  of  models  in  the  acquisition  process.  ASN(RDA)  is  a  member  of  the 
DON  M&S  Oversight  Council.  The  point  of  contact  (POC)  is: 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

The  Pentagon 

Washington,  DC  20350-1000 
Phone:  (703)  695-2843 
DSN:  225-2843 


Chief  of  Naval  Operations,  Space  and  Electronic  Warfare  Directorate  (N6) 

The  Navy’s  Policy  and  Coordination  Office  for  Modeling  &  Simulation  is  expected  to  be 
established  in  this  directorate  during  the  summer  of  1994.  The  specific  office  had  not  been 
determined  as  of  May  1994.  The  POC  is: 

Chief  of  Naval  Operations  (N6) 

The  Pentagon 

Washington,  DC  20350-2000 
Phone:  (703)  695-4379 
DSN:  225-4379 
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Deputy  Chief  of  Naval  Operations  (Resources,  Warfare  Requirements  and  Assessment), 
Assessment  Division,  Assessments  and  Affordability  Branch,  Modeling  and  Analysis  Sec¬ 
tion  (N812D) 

This  office  is  the  focal  point  for  Navy  modeling  and  simulation.  The  Navy  M&S  function  is 
expected  to  transition  to  N6  during  the  summer  of  1994.  The  POC  is: 

Chief  of  Naval  Operations  (N812D) 

The  Pentagon 

Washington,  DC  20350-2000 
Phone:  (703)  697-5242 
DSN:  227-5242 


Space  and  Naval  Warfare  Systems  Command  (SPAWAR  31) 

SPAWAR  31  leads  the  Navy’s  M&S  Technical  Support  Group.  The  POC  is; 

Commander,  Space  and  Naval  Warfare  Systems  Command,  (SPAWAR  31) 

2451  Crystal  Drive 
Arlington,  VA  22245-5200 
Phone:  (703)  602-4540 
DSN:  332-4540 


USERS  AND/OR  DEVELOPERS 

The  following  are  users  and/or  developers  of  models  and  simulations.  There  may  be  a  num¬ 
ber  of  groups  within  each  organization  which  deal  with  M&S.  However,  a  single  point  of 
contact  (code)  is  listed  for  each  organization. 

Naval  Air  Systems  Command  POC  is: 

Commander,  Naval  Air  Systems  Command 
(PMA-205) 

1421  Jefferson  Davis  Highway 
Arlington,  VA  22243 
Phone:  (703)  604-2245,  ext.3046 
DSN:  664-2245  ext.  3046 
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Naval  Air  Warfare  Center/Aircraft  Division  (Warminster)  POC  is; 


Commander,  Naval  Air  Warfare  Center/ Aircraft  Division  (Code  30B) 

PO.  Box  5152 
Warminster,  PA  18974 
Phone;  (215)  441-1534 
DSN:  441-1534 


Naval  Air  Warfare  Center/ Aircraft  Division  (Patuxent  River)  POC  is: 


Naval  Air  Warfare  Center/Aircraft  Division 
Flight  Test  and  Engineering  Group  (Code  SY02C) 
Patuxent  River,  MD  20670 
Phone:  (301)  826-6009 
DSN:  326-6009 


Naval  Air  Warfare  Center/TVaining  Systems  Division  POC  is: 

Commander,  Naval  Air  Warfare  Center/Training  Systems  Division 

(Code  PDB8) 

12350  Research  Parkway 
Orlando,  FL  32826 
Phone:  (407)  380-8367 
DSN:  960-8367 


Naval  Air  Warfare  Center/Weapons  Division  POC  is; 

Commander,  Naval  Air  Warfare  Center/Weapons  Division 
(Code  C0243) 

China  Lake,  CA  93555 
Phone:  (619)  939-2353 
DSN:  437-2353 


Naval  Sea  Systems  Command  POC  is: 

Commander,  Naval  Sea  Systems  Command 
(SEA-91  Wl,  Combat  Systems  Training) 
2351  Jefferson  Davis  Highway 
Arlington,  VA  22242-5160 
Phone;  (703)  602-1782 
DSN:  332-1782 
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Naval  Surface  Warfare  Center/Headquarters  POC  is: 


Commander,  Naval  Surface  Warfare  Center 
(NSWC-04M) 

2531  Jefferson  Davis  Highway 
Arlington,  VA  22242-5160 
Phone:  (703)  602-0632 
DSN;  332-0632 


Naval  Surface  Warfare  Center/Carderock  Division  POC  is: 

Naval  Surface  Warfare  Center/Carderock  Division 
(Code  21) 

Bethesda,  MD  20084-5000 
Phone:  (301)  227-1013 
DSN:  287-1013 


Naval  Surface  Warfare  Center/Crane  Division  POC  is: 

Naval  Surface  Warfare  Center/Crane  Division 
(Code  604) 

Bldg.  2045 
300  Highway  361 
Crane,  IN  47522-5001 
Phone:  (812)  854-3966 
DSN;  482-3966 


Naval  Surface  Warfare  Center/Dahlgren  Division  POC  is: 


Naval  Surface  Warfare  Center/Dahlgren  Division 
(Code  AOS) 

Dahlgren,  VA  22448-5000 
Phone:  (703)  663-7369 
DSN:  249-7369 
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Naval  Surface  Warfare  Center/Dahigren  Division/Coastai  Systems  Station  POC  is 


Commanding  Officer,  Naval  Surface  Warfare  Center/ 
Dahlgren  Division/Coastal  Systems  Station 
(Code  04) 

Panama  City,  FL  32407 
Phone:  (904)  234-4200 
DSN:  436-4200 


Naval  Surface  Warfare  Center/Indian  Head  Division  POC  is: 

Naval  Surface  Warfare  Center/Indian  Head  Division 


(Code  64C4) 

Indian  Head,  MD  20640-5000 
Phone:  (301)  743-4397 
DSN:  354-4397 


Naval  Surface  Warfare  Center/Port  Hueneme  Division  POC  is: 

Naval  Surface  Warfare  Center/Port  Hueneme  Division 
(Code  4L12) 

4363  Missile  Way 
Port  Hueneme,  CA  93043-4307 
Phone:  (805)  982-7023 
DSN:  551-7023 


Hydrodynamics/Hydroacoustics  Technology  Center  (NAVSEA-03HT)  POC  is: 


Carderock  Division,  NSWC 
Building  17,  Room  120 
Bethesda  MD  20084-5000 
Phone:  (301)  227-3827 
DSN:  287-3827 


C-5 


r 


Naval  Command  &  Control  and  Ocean  Surveillance  Center/RDT&E  Division  POC  is: 

Commander,  Naval  Command  &  Control  and  Ocean  Surveillance  Center/ 

RDT&E  Division 
(Code  78) 

San  Diego,  CA  92152 
Phone:  (619)  553-1637 
DSN:  553-1637 


Naval  Undersea  Warfare  Center  POC  is: 

Commander,  Naval  Undersea  Warfare  Center  Detachment,  New  London 

(Code  63) 

39  Smith  St. 

New  London,  CT  06320 
Phone:  (203)  440-4059 
DSN:  636-4059 


Naval  Research  Laboratory  POC  is: 

Commander,  Naval  Research  Laboratory 
(Code  5550) 

4555  Overlook  Drive,  S.W., 
Washington,  DC  20375 
Phone:  (202)  767-3162 
DSN:  297-3162 


COMOPTEVFOR  POC  is: 


COMOPTEVFOR 
(Code  332) 

7970  Diven  St. 
Norfolk,  VA  23505-1498 
Phone:  (804)  445-0292 
DSN:  565-0292 
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Center  for  Naval  Analysis  POC  is: 

Center  for  Naval  Analysis 
4401  Ford  Avenue 
PO.  Box  16268 
Alexandria,  VA  22302-0268 
Phone:  (703)  824-2352/2998 
DSN:  289-2352 


Naval  Center  for  Cost  Analysis  POC  is: 

Naval  Center  for  Cost  Analysis 
1111  Jefferson  Davis  Highway 
Suite  400  West  Tower 
Arlington  VA  22202-4306 
Phone:  (703)  604-0293 
DSN:  664-0293 


NAVY  MODELING  AND  SIMULATION  CATALOG 


The  Navy  Modeling  and  Simulation  Catalog  contains  information  on  a  number  of  Navy 
models  ranging  from  engineering  to  campaign  level.  The  catalog  focuses  on  models  used 
for  Joint  Mission  Area  and  Support  Area  Assessments  performed  by  the  Chief  of  Nava) 
Operations.  The  POC  is: 

Space  and  Naval  Warfare  Systems  Command 
(SPAWAR  312) 

2451  Crystal  Drive 
Arlington,  VA  22245-5200 
Phone:  (703)  602-4541 
DSN:  332-4541 
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DEPARTMENT  OF  THE  NAVY  DOCUMENTS 


In  addition  to  the  DON  Instructions  on  Modeling  and  Simulation  described  in  Chapter  3, 
the  following  documents  concerning  acquisition  and  M&S  may  be  useful  to  the  reader. 


OPNAV  Instruction  5(H)0.42D.  (19  April  1993).  OPNAV  Role  and  Responsibilities  in  the 
Acquisition  Process. 


SECNAVINST  5000.2A.  (9  Dec  92).  Implementation  of  Defense  Acquisition  Management 
Documentation  and  Reports. 


PRINCIPLES  FOR  VERIFICATION,  VALIDATION,  AND  ACCREDITATION  (W&A)  OF 
NAVY  MANAGED  MODELS  AND  SIMULATIONS  (M&S),  dated  15  December  1993, 
prepared  by  SPAWAR  31.  This  document  identifies  principles  and  techniques  to  enhance 
V&V  of  Navy  managed  models  and  simulations  and  describes  methods  to  facilitate  ac¬ 
creditation.  It  describes  three  levels  of  accreditation  and  discusses  the  requirements  for 
each  level,  including  model  documentation,  verification  plans  &  tests,  validation  plans  & 
tests,  configuration  management,  data,  and  personnel  qualifications. 
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APPENDIX  D 

MARINE  CORPS  SOURCES  OF  INFORMATION 
f  FOR  MODELING  AND  SIMULATION 

IN  WEAPON  SYSTEMS  ACQUISITION 


The  information  contained  in  this  appendix  draws  heavily  upon  the  list  of  organizations 
contained  within  the  Marine  Corps  Modeling  and  Simulation  Master  Plan  (Draft,  dated  16 
March  1994. 


MARINE  CORPS  MODELING  AND  SIMULATION 
MANAGEMENT  OFFICE  (MCMSMO) 

The  MCMSMO  is  the  focal  point  for  modeling  and  simulation  (M&S)  in  the  Marine  Corps. 
It  serves  as  the  management  and  coordinating  activity  for  all  M&S  related  activities  and 
provides  limited  technical  support  to  M&S  users.  The  MCMSMO  supports  development 
of  and  manages  the  Marine  Corps  Modeling  and  Simulation  Master  Plan  and  long  range 
investment  strategies.  The  point  of  contact  (POC)  is: 

MCCDC,  T&E  Div,  MCMSMO  (Code:  C46MS) 

2006  Hawkins  Ave 
Quautico,  VA  22134 
Phone:  (703)  640-2520 


lyaining  and  Education  (T&E)  Division,  MCCDC:  The  T&E  division  is  responsible  for  the 
development  and  implementation  of  policy  and  programs  for  training  and  education  of  all 
regular  and  reserve  Marine  Corps  personnel  and  units.  They  function,  among  other  things, 
as  the  proponent  for  non-standard  training  devices  and  simulators,  validate  training  device 
requirements,  and  develop  and  refine  the  Marine  Corps  Ground  Range  Program.  This 
division  also  functions  as  the  proponent  for  MTWS  and  provides  constructive  modeling 
support  to  the  Fleet  Marine  Force  (FMF),  Marine  Corps  University  (MCU)  and  other 
agencies.  The  POC  is: 


MCCDC,  T&E  Div  (Code:  C46) 
1019  Elliot  Rd. 
Quantico,  VA  22134 
Phone:  (703)  640-3731 


Marine  Corps  Operational  Test  and  Evaluation  Activity  (MCOTEA):  The  MCOTEA  is 
the  Marine  Corps  principal  operational  test  organization.  The  MCOTEA  is  an  active  mem¬ 
ber  of  the  MCMSWG  and  provides  the  necessary  test  perspective  to  Marine  Corps  M&S 
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policies,  procedures  and  guidelines.  The  POC  is: 


MCCDC,  MCOTEA 
3035  Barnett  Ave 
Quantico,  VA  22134 
Phone;  (703)  640-3141 


Marine  Corps  Tactical  Systems  Support  Activity  (MCTSSA),  Marine  Corps  Systems  Com¬ 
mand  (MARCORSYSCOM):  This  activity  supports  tactical  software  development  by  con¬ 
ducting  testing  and  providing  maintenance  through  the  software  life  cycle.  The  MCTSSA’s 
integration  of  M&S  in  its  testing  process  is  a  critical  component  in  accomplishing  many  of 
the  Marine  Corps  M&S  end  states.  The  POC  is: 

MARCORSYSCOM 

MCTSSA 

2033  Barnett  Ave,  Suite  315 
Quantico,  VA  22134-5010 
Phone:  (703)  640-3792 
DSN;  278-2411 


Amphibious  Warfare  Technology  Directorate  (AWT),  MARCORSYSCOM:  Research  and 
demonstration  of  future  technologies  and  their  applicability  to  the  Corps  is  the  responsibil¬ 
ity  of  AWT.  A  significant  element  in  AWT  activities  is  M&S  related  technology.  The  AWT 
also  provides  technology  insights  which  support  MCMSWG  activities.  The  POC  is: 

MARCORSYSCOM 

AWT 

2033  Barnett  Ave,  Suite  315 
Quantico,  VA  22134-5010 
Phone:  (703)  640-3792 
DSN:  278-2411 


Wargaming  and  Combat  Simulation  Division  (WCSD):  The  WCSD  provides  wargaming 
and  assessment  support  for  HQMC,  the  operating  forces  (active  and  reserve),  and  the 
supporting  establishment  including  the  Marine  Corps  Combat  Development  Command 
(MCCDC).  The  POC  is; 


MCCDC,  WCSD,  (Code:  C471) 
2076  South  St. 
Quantico,  VA  22134 
Phone:  (703)  640-3276 
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MAGTF/Expeditionary  Ti^ining  Center  (M/ETC):  The  M/ETC  mission  is  to  strengthen 
and  improve  the  coordination  and  integration  of  training  opportunities  among  MCAGCC, 
MAWTS,  MWTC,  LFTCs  and  other  Service  Training  Centers  in  a  Naval  Expeditionary 
Warfare  context.  The  M/ETC  will  compliment  training  through  the  use  of  models  and  simu¬ 
lators  that  support  mission  preview  and  rehearsal,  and  battle  staff  training.  Additionally, 
M/ETC  will  use  the  Global  Grid  to  participate  in  Joint  and  CINC  level  exercises  and  in¬ 
crease  training  opportunities  for  geographically  dispersed  active  and  reserve  MAGTF  ele¬ 
ments.  The  POC  is: 

MCCDC,  MSTP  (C466) 

2024  Barnett  Ave,  Suite  202 
Quantico,  VA  22134 
Phone:  (703)  640-3279 
DSN:  278-3279 

Marine  Corps  Computer  and  Telecommunications  Activity  (MCCTA),  Headquarters  Ma¬ 
rine  Corps  (HQMC)  (C4I2):  The  MCCTA  is  a  major  activity  pursing  and  implementing 
Marine  Corps  requirements  relating  to  establishing  the  Global  Grid.  Additionally,  MCCTA 
participates  in  developing  M&S  verification,  validation  and  accreditation  (VV&A)  and 
configuration  management  policies,  procedures  and  guidelines  as  a  member  of  the  Marine 
Corps  Modeling  and  Simulation  Working  Group  (MCMSWG).  The  POC  is: 

CMC,  HQMC 
MCCTA 
2  Navy  Annex 

Washington,  DC  20380-1775 
Phone:  (703)  614-2443/2604 
DSN:  224-2443/2604 

ADS  Demonstration  Sites:  Two  ADS  demonstration  sites,  the  MCAGCC  ADS  Demon¬ 
stration  Site  located  at  TVenty-nine  Palms  and  the  proposed  DMSC  at  Quantico  provide 
an  operational  setting  to  demonstrate  and  test  M&S  components.  The  POC  is: 

MCCDC,  T&E  Div,  MCMSMO  (Code:  C46MS) 

2006  Hawkins  Ave. 

Quantico,  VA  22134 
Phone:  (703)  640-2498/2520 
DSN:  278-2498 
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Program  Support  Directorate  (PS),  MARCORSYSCOM:  The  PS  directorate  is  a  matrix 
organization  supporting  the  program  manager  (PM)  in  engineering,  logistics,  technical 
manual  documentation,  operations  analysis  and  life  cycle  cost  analysis.  The  PS  has  been 
designated  the  focal  point  for  M&S  within  MARCORSYSCOM.  The  POC  is; 

MARCORSYSCOM 

Program  Support  Directorate  (Code:  PSA) 

2033  Barnett  Ave.,  Suite  315 
Quantico,  VA  22134 
Phone:  (703)  640-4451 
DSN:  278-4451 
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APPENDIX  E 

DEPARTMENT  OF  THE  AIR  FORCE  SOURCES 
OF  INFORMATION  FOR  MODELING  AND  SIMULATION 
IN  WEAPON  SYSTEMS  ACQUISITION 

This  appendix  provides  information  regarding  various  Air  Force  organizations’  M&S 
activities.  It  provides  office  symbols,  phone  numbers  and  addresses  to  assist  users  in 
making  initial  contact. 

POLICY  AND  GUIDANCE 

Headquarters  U.S.  Air  Force,  Directorate  of  Modeling,  Simulations,  and  Analysis  (HQ 

USAF/XOM):  The  HO  USAF/XOM  promulgates  Air  Force  M&S  policy;  publishes  guid¬ 
ance  and  administers  the  development  of  AFPD  16-10,  Modeling  and  Simulation  Manage¬ 
ment  {dratt,  May  1994),  AFI  16-1001,  Verification,  Validation,  and  Accreditation  (draft),  and 
AFI  16-1002,  M&S  Management  (draft).  The  HQ  USAF/XOM  is  the  single  point  of  contact 
for  M&S  issues  and  activities  within  the  Air  Force.  They  also  represent  the  Air  Force  in 
joint,  multi-service  and  multi-agency  M&S  efforts. 

POINTS  OF  CONTACT: 

This  list  is  a  continuation  of  very  good  information  provided  in  AFMCP  800-66,  Atch  7, 
dated  1  July  1993.  This  pamphlet  is  also  a  good  reference  for  specific  model  information; 
description,  OPR  and  phone  numbers.  This  list  is  not  a  complete  listing  of  all  M&S  con¬ 
tacts,  however,  it  is  a  good  starting  point  in  locating  assistance. 


HQ  USAF/XOM  Room  4C1059 

Phone: 

(703)  695-1835/1847 

1400  Air  Force,  Pentagon 

DSN; 

225-1847 

Washington  D.C.  20330-1400 

FAX: 

(703)  695-1161 

HQ  USAF/XOME 

Phone: 

(202)  504-5333 

Evaluation  Support  Division 

624  9th  Street  NW,  Suite  300 

Washington  D.C.  20001-6303 

DSN. 

285-5333 

HQ  USAF/XOMT 

Phone: 

(202)  504-5339 

Technical  Support  Division 
(same  address  as  XOME) 

DSN; 

285-5339 

HQ  USAF/XOMW 

Phone: 

(202)  504-4441 

Warfighting  Support  Division 

DSN: 

285-4441 

(same  address  as  XOME) 
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AFSAA  Room  1E388 

Air  Force  Studies  and  Analysis 

1400  Air  Force,  Pentagon 

Washington  D.C.  20330-1400 

Phone; 

DSN; 

(703)  695-9048 
225-9048 

OTHER  ORGANIZATIONS  WITH  M&S  EXPERIENCE: 

HQ  AFMC/XRX 

Director  for  Requirements 

4375  Chidlaw  Rd.  Suite  6 

Wright-Patterson  AFB,  OH  45433-5001 

Phone; 

DSN; 

(513)  257-4914 
787-4914 

HSC/XR  (AFMC) 

Human  Systems  Center 

Brooks  AFB  TX  78235-5000 

Phone; 

DSN; 

(512)  471-3406 
240-3406 

SMC/XR  (AFMC) 

Space  &  Missile  Systems  Center 

Los  Angeles  AFB  CA  90009-2960 

Phone; 

DSN: 

(310)  336-4613 
833-4613 

ESC/XRP  (AFMC) 

Electronic  Systems  Center 

Hanscom  AFB  MA  01731-5000 

Phone: 

DSN: 

(617)  377-6554 
478-6554 

ASC/XR  (AFMC) 

Aeronautical  Systems  Center 
Wright-Patterson  AFB  OH  45433-5000 

Phone: 

DSN: 

(513)  255-4656 
785-4656 

Armstrong  Laboratory  (AFMC) 

Brooks  AFB  TX  78235-5000 

Phone: 

DSN: 

(512)  471-2424 
240-2424 

Phillips  Laboratory  (AFMC) 

Kirtland  AFB  NM  87117-6008 

Phone: 

DSN: 

(805)  846-1737 
246-1737 

Rome  Laboratory  (AFMC) 

Griffiss  AFB  NY  13441-5700 

Phone; 

DSN: 

(315)  330-7701 
587-7701 

Wright  Laboratory  (AFMC) 

Wright-Patterson  AFB  OH  45433-6553 

Phone: 

DSN; 

(513)  255-4840 
785-4840 

OAS/XR 

Office  of  Aerospace  Studies 

Kirtland  AFB,  NM  87117-6008 

Phone: 

DSN: 

(505)  846-8322 
246-8322 
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AFSPACECOM/CNA 

HQ  Air  Force  Space  Command 

Peterson  AFB  CO  80914-5001 

Phone: 

DSN: 

(719)  554-5196 
692-5196 

AFIC/DOA 

Air  Force  Intelligence  Cmd 

Kelly  AFB  TX  78243-5000 

Phone; 

DSN: 

(210)  977-2877 
969-2877 

AFCC/XR 

HQ  Air  Force  Communications  Cmd 

Scott  AFB  IL  62225-5000 

Phone: 

DSN: 

(618)  256-5541 
576-5541 

ACC/XP-JSG 

HQ  Air  Combat  Command 

Langley  AFB  VA  23665-5520 

Phone; 

DSN; 

(804)  764-5751 
574-5751 

AMC/XPY 

HQ  Air  Mobility  Command 

Scott  AFB  IL  62225-5001 

Phone: 

DSN: 

(618)  256-5560 
576-5560 

ATC/XPC 

Air  Training  Command 

Randolph  AFB  TX  78150-5000 

Phone: 

DSN: 

(210)  652-2640 
487-2640 

AFOTEC/SAN 

AF  Operational  Test  &  Eval  Ctr 

Kirtland  AFB  NM  87117-6008 

Phone: 

DSN: 

(505)  846-1357 
246-1357 

AWS/XTX 

Air  Weather  Service 

Scott  AFB  IL  62225-5008 

Phone; 

DSN: 

(618)  256-4598 
576-4598 

STRATCOM/J53 

HQ  US  Strategic  Command 

Force  Assessment  Division 

Offutt  AFB  NE  68113-5001 

Phone: 

DSN: 

(402)  294-2355 
271-2355 

AFDTC/XRP 

AF  Developmental  Test  Center 

Eglin  AFB,  FL  32542-5495 

Phone: 

DSN; 

(904)  882-4188 
872-4188 

AFFTC/XRX 

AF  Flight  Test  Center 

Edwards  AFB  CA  93542-1036 

Phone: 

DSN: 

(805)277-3837 

527-3837 
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AEDC/DOT 

Arnold  Engineering  Development  Ctr 
Arnold  AFB  TN  37389-9011 

Phone; 

DSN: 

(615)454-6508 

340-6508 

HQ  AFMC/STXP 

AFMC  Science  & 

Technology  Directorate 

Wright-Patterson  AFB  OH  45433-5006 

Phone: 

DSN: 

(513)257-7850 

787-7850 

AFOSR/XPP 

Air  Force  Office  of 

Scientific  Research 

Bolling  AFB  DC  20332-0001 

Phone; 

DSN: 

(202)767-6010 

297-6010 

HQ  AFMC/LGP 

AFMC  Logistics  Directorate 
Wright-Patterson  AFB  OH  45433-5006 

Phone: 

DSN: 

(513)257-5610 

787-5610 

OC-ALC/FMPM 

Oklahoma  City  Air  Logistics  Center 

Tinker  AFB  OK  73145-3056 

Phone: 

DSN; 

(405)739-2519 

336-2519 

OO-ALC/FMPM 

Ogden  Air  Logistics  Center 

Hill  AFB  UT  84056-5038 

Phone: 

DSN: 

(801)777-5851 

924-5851 

SA-ALC/FMPB 

San  Antonio  Air  Logistics  Center 

Kelly  AFB  TX  78241-6435 

Phone: 

DSN: 

(201)925-6726 

945-6726 

SM-ALC/FMPM 

Sacramento  Air  Logistics  Center 

McClellan  AFB  CA  95652-1060 

Phone: 

DSN: 

(916)643-6162 

633-6162 

WR-ALC/FMPX-1 

Warner  Robins  Air  Logistics  Center 

Robing  AFB  GA  31098-1640 

Phone: 

DSN: 

(912)926-3202 

468-3202 

AGMC/FM(2) 

Aerospace  Guidance  &  Metrology  Ctr 

Phone; 

DSN: 

(614)522-7643 

346-7643 

Newark  AFB  OH  43057-5260 
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CATALOGS/SOURCE  DOCUMENTS: 


Catalog  of  Simulation  Models  and  Wargames  Used  for  Unit  and  Leader  Training.  (1987).  (2nd 
ed.)  Orlando  FL:  Training  and  Performance  Data  Center. 

Catalog  of  War  Games,  Training  Games,  and  Combat  Simulations.  (19S3).  Washington  DC; 
Deputy  Under  Secretary  of  the  Army  (Operations  Research). 

Department  of  Defense  Catalog  of  Logistics  Models.{1990).  Fort  Lee  VA;  Defense  Logistics 
Studies  Information  Exchange. 

1990  Catalog  of  Computer  Simulation  Tools.{l99Q).  Washington  DC:  Air  Force  Center  of 
Studies  and  Analysis. 

Major  Military  Models  Written  in  SIMSCRIPT  11.5.  (3rd  ed.)  La  Jolla  CA:  C.A.C.I. 

A  Summary  of  Analysis  Methodolo^es  Used  in  the  Directorate  of  Mission  Analysis,  (1987). 
Wright-Patterson  AFB  OH:  Deputy  for  Development  Planning,  Aeronautical  Systems 
Center. 

Survey  of  Models/Simulations  at  RL  (Rome  Laboratory).  (1986).  Vol.  3.  Griffiss  AFB  NY: 
Rome  Air  Development  Center. 

Science  Advisory  Board  (SAB)  ad  hoc  Committee  on  Modeling  and  Simulation,  December 
1991. 

Science  Advisory  Board  (SAB)  Repo:  for  Theater  Missile  Defense  (TMD),{\99y). 

AFMCP  800-66.  (July  1993).  AFMC  Modeb  and  Simulations  (M&S)  Guide.  Wright-Patterson 
AFB,  OH:  HQ  Air  Force  Materiel  Command. 

HQ  USAF/XOM.  Newsletter:  Issues  in  Air  Force  Simulation  and  Analysis,  (January  1994). 
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APPENDIX  F 

ADDITIONAL  SOURCES  OF  INFORMATION  FOR  MODELING 
AND  SIMULATION  IN  WEAPON  SYSTEMS  ACQUISITION 


This  appendix  provides  information  on  the  Modeling  and  Simulation  Information  System 
(MSIS)  and  the  DoD  Information  Analysis  Centers  (lAC).  These  are  additional  sources 
of  information  which  may  assist  program  managers  in  their  search  for  models,  simulat'ons 
or  data. 

The  Modeling  and  Simulation  Information  System 

1.  Name  of  System;  Modeling  &  Simulation  Information  System  (MSIS). 

2.  Proponent/Sponsor/Operator:  Defense  Modeling  &  Simulation  Office  (DMSO). 

3.  Purpose:  Serve  as  an  information  clearing  house  for  M&S  and  related  information. 
Provide  a  means  of  disseminating  and  coordinating  M&S  related  information. 

4.  System  Contents;  Menu  driven  system  containing  catalogs  to  provide  summary  in¬ 
formation  on  models,  simulations  and  applicable  data  (including  the  J-8  and  Service  spe¬ 
cific  catalogs);  documents  and  reports  pertaining  to  activities  and  developments  in  M&S 
along  with  related  and  supporting  areas;  glossary  of  terms;  calendar  of  events;  remote 
access  to  other  M&S  information  repositories.  Key  word/topic  searci»  capability  in  cata¬ 
logs  and  other  modules.  Al!  data  are  unclassified,  with  unrestricted  distribution.  The  sys¬ 
tem  also  provides  a  number  of  support  utilities  and  functions,  including  e-mail. 

5.  Account  Availability:  Individuals  with  interests  in  Modeling  &  Simulation  and  re¬ 
lated  areas. 

6.  Requirements  for  Use;  Users  must  register  (see  POC  information,  below)  to  obtain  an 
account  on  the  system.  User  must  provide  necessary  hardware  and  software  to  communi¬ 
cate  with  the  system,  e.g.,  PC  with  modem  and  terminal  emulation/communications  soft¬ 
ware  or  woikstation  with  internet  access.  Direct  dial  (703  area  code)  and  tymnet  (local 
number  access,  nation-wide)  access  are  provided.  Users  may  also  connect  over  the  internet 
via  telnet.  Users  with  systems  running  a  gopher  client  may  connect  to  the  system  through 
the  gopher  client.  As  of  July  1994  there  is  no  charge  for  connect  time. 

7.  Point  of  Contact  (POC)  for  information  or  user  accounts: 

Modeling  and  Simulation  Information  System 
ATTN:  Administrative  Supr 
1901  N.  Beauregard  St.,  Suite  510,  Alexa.  vA  22311 
Phone;  (703)  379-3770  Fax:  (703 j  v-3778 

E-mail:  dmso@dmso.dtic.dla.mil  -  e-mail 
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CATALOGS  LISTED  IN  THE  MSIS  (AS  OF  JULY  1994): 


1.  J-8  Catalog  of  Models  and  Simulations  (12th  edition) 

2.  TRANSCOM  System  Model  Catalog  (Sep  89) 

3.  Catalog  of  War  Games,  Training  Games  and  Combat  Simulation 

4  Models  and  Simulations:  Army  Integrated  Catalog  (MOSAIC)  (Dec  93) 

5.  Navy’s  Catalog  of  Models  and  Simulations  (5  Oct  93) 

6.  U.S.  Air  Force  Rome  Laboratory  Models  and  Simulations  Catalog  (Oct  93) 

7.  DMSO  -  Catalogs  of  Models  and  Simulations 

8.  Institute  for  Simulation  and  Training  (1ST)  Catalog  of  Models  and  Simulations  Docu¬ 
ments,  Videos,  Files,  etc. 

NOTE:  The  content  and  format  of  the  entries  in  the  individual  catalogs  vary. 


THE  DoD  INFORMATION  CENTER  ANALYSIS  CENTERS 

The  DoD  lACs  provide  information  to  users  which  allows  them  to  benefit  from  experi¬ 
ences  of  counterparts  in  comparable  fields  of  endeavor,  and  increase  productivity  and  quality 
of  research. 

The  lACs  are  established  under  r)op  Regulation  3200.12-R-2,  Centers  for  the  Analysis  of 
Scientific  and  Technical  Information,  dated  17  January  1985.  Their  primary  mission  is  to 
collect,  analyze,  .synthesize  and  disseminate  worldwide  scientific  and  technical  information 
in  clearly  defined,  specialized  fields  or  subject  areas. 

There  are  26  lACs  supported  by  DoD,  maintaining  comprehensive  knowledge  bases  which 
include  historical,  technical,  scientific  and  other  data  and  information  collected  on  a 
worldwide  basis.  Information  collections  include  a  wide  range  of  unclassified,  limited  dis¬ 
tribution  and  classified  information  appropriate  to  the  requirements  of  sponsoring  techni¬ 
cal  communities. 

The  lACs  also  collect,  maintain,  and  develop  analytical  tools  and  techniques  including  data 
bases,  models  and  simulations.  Their  collections  and  products  represent  intensive  evaluation 
and  screening  efforts  to  create  authoritative  sources  of  evaluated  data. 

For  the  purposes  of  this  guide,  the  subject  matter  covered  by  existing  DoD  lACs  is  related 
to  military  functions,  roles/missions  and  to  key  technologies.  Figures  F-1  and  F-2  display 
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lACs  Associated  with  Military  Missions/Functions 


Military  Missions/Functional  is  ll 

Area  of  Potential  User  =  ^  1 1  ^  .  I  ° 


APMIAC  (Airfields,  Pavements  &  Mobility) 

OnHOEl 

■HJIOOHl 

OOODO 

CBIAC  (Chemical/Biological  Warfare) 

CEIAC  (Coastal  Engineering) 

ODHO 

CIAC  (Ceramics) 

■iOOKJ 

CPIA  (Chemical  Propulsion) 

aaaaa 

DO 

1 

CRISTIAC  (Cold  Regions) 

DODDD 

D 

CSERIAC  (Crew  Systems  Ergonomics) 

OOOH 

O 

in 

CTIAC  (Concrete  Technology) 

EJiUnHI 

El 

DACS  (Data  Analysis  Center  for  Software) 

OODDDOO 

DASIAC  (Nuclear  Effects) 

ODDDH 

D 

a 

GACIAC  (Guidance  &  Control) 

onnaao 

D 

HEIAC  (Hydraulic  Engineering) 

DOHHI 

D 

HTMIAC  (High  Temperature  Materials) 

DonnH 

D 

IRIA  (Infrared) 

nnon 

DODO 

— 

DO 

— 

MIAC  (Metals) 

MMCIAC  (Metal  Matrix  Composites) 

ODODHI 

D 

MTIAC  (Manufacturing  Technology) 

ITZ 

D 

D 

D 

NTIAC  (Nondestructive  Testing) 

DDODD 

D 

D 

PLASTEC  (Plastics) 

RAC  (Reliability) 

■DDOODOO 

SMIAC  (Soil  Mechanics) 

KUHiDEl 

Kimwjima 

DOHIDHi 

ODDDD 

SURVIAC  (SurvivabilityAfulnerability) 

DDDD 

DDDD 

TWSTIAC  (Tactical  Warfare  Simulation  and  Technology) 

Figure  F-1.  lACs  Associated  with  Military  Missions/Functions 


the  association  of  each  lAC  based  on  this  organization. 

Figure  F-1  maps  DoD  lACs  for  users  based  on  military  mission  or  function.  The  lACs  are 
identified  by  their  abbreviations,  which  will  be  expanded  later  in  this  appendix. 

For  example,  a  user  involved  in  planning,  analysis  or  acquisition  in  support  of  Special  Op¬ 
erations  Forces  (SOF)  will  find  lACs  with  a  •  in  the  column  labeled  “SOF”  more  likely 
(but  not  exclusive)  providers  of  appropriate  and  useful  information  products  and  services. 
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Figure  F-2.  lACs  Associated  with  DoD  Key  Technologies 


Although  likely  to  find  useful  M&S  products  within  the  TWSTIAC,  a  user  would  also  pos¬ 
sibly  find  useful  M&S  products  addressing  survivability  within  the  SURVIAC. 


Figure  F-2  illustrates  the  subject  matter  focus  of  lACs  as  they  relate  to  DoD  Key  Technolo¬ 
gies.  Again,  a  •  in  a  Key  Technology  column  highlights  areas  of  special  interest  or  focus 
within  the  lAC  program.  A  key  feature  of  the  DoD  lACs  is  that  they  work  together  to  ensure 
that  user  needs  are  met. 
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FEES  FOR  SERVICES 


Financial  support  for  basic  lAC  operations  is  provided  by  the  Defense  Technical  Informa¬ 
tion  Center  (DTIC).  However,  the  lACs  offset  costs  incurred  in  collecting,  analyzing  and 
disseminating  information  by  a  service  charge  structure  implemented  in  accordance  with 
guidance  provided  by  DoD.  However,  no  charges  are  incurred  without  the  explicit  agreement 
of  the  customer. 


CONTACTING  THE  DoD  lACs 

Users  may  contact  the  appropriate  lAC(s)  directly  for  specific  information.  For  general 
information  on  the  DoD  lAC  program,  or  for  assistance  in  locating  an  lAC  to  service  a 
particular  need,  users  can  contact: 

Defense  Technical  Information  Center 
ATTN:  DTIC-AI 
Cameron  Station 
Alexandria,  VA  22304-6145 
Phone:  (703)274-6260 
DSN:  284-6260 
FAX:  (703)274-0980 
E-mail:  iac@dgis.dtic.dla.mil 


ALPHABETIC  LISTING  OF  DoD  lACs 

Airflelds,  Pavements  and  Mobility  Information  Analysis  Center  (APMIAC): 

Subject  coverage  -  Airfields,  pavements,  vehicle  mobility,  and  terrain,  as  relevant  primarily 
to  military  needs.  Specific  areas  of  vehicle  on-  and  off-road  mobility,  ground  flotation  and 
terrain  evaluation.  The  POC  is: 


Commander  and  Director 

U.S.  Army  Engineer  Waterways  Experimentation  Station 
ATTN.  CEWES/GM-L 
3909  Halls  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone:  Tech/Biblio  Inquiries: 
Mobility/Terrain  -  (601)634-2734 
Airfields/Pavements  -  (601)634-3304 
Computerized  Database/Library  -  (601)634-4120 
FAX:  (601)634-3068 
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Ceramics  Information  Analysis  Center  (CIAC) 

Subject  coverage  -  Source  of  engineering  and  technical  data,  and  research  and  develop¬ 
ment  information  on  monolithic  ceramics  and  ceramic  composites,  hybrids,  laminates  and 
coatings  used  in  Defense  systems.  The  POC  is; 

CIAC/CINDAS 
Purdue  University 
2595  Yeager  Road 
West  Lafayette,  IN  47906-1398 
Phone:  (317)494-9393 
FAX:  (317)496-1175 

Chemical  Propulsion  Information  Agency  (CPIA) 

Subject  coverage  -  Acquisition,  analysis  and  dissemination  of  information  in  the  areas  of 
missile,  space  and  gun  propulsion  technology.  Hardware  of  interest  includes  rocket  mo¬ 
tors,  rocket  engines,  air  breathing  missile  propulsion  systems,  electric  and  nuclear  space 
propulsion  systems,  electric  and  conventional  guns,  gas  generators,  mines  and  torpedoes, 
and  their  inert  and  energetic  components.  The  POC  is: 

The  Johns  Hopkins  University 
Chemical  Propulsion  Information  Agency 
10630  Little  Patuxent  Parkway,  Suite  202 
Columbia,  MD  21044-3201 
Phone:  (410)992-7306 
FAX:  (410)730-4969 

Chemical  Warfare/Chemical  and  Biological  Defense  (CW/CBD)  Information  Analysis 
Center  (CBIAC): 

Subject  coverage  -  DoD  focal  point  for  technical  information  related  to  CW/CBD.  The 
CBIAC  collects,  analyzes,  summarizes  and  stores  CW/CBD  information  available  from 
both  domestic  and  foreign  sources.  The  POC  is: 

Battelle  Edgewood  Operations 
ATTN:  CBIAC 

2113  Emmorton  Park  Road,  Suite  200 
Edgewood,  MD  21040-1037 
Inquiries,  Products,  Publications:  (601)634-2734 
FAX:  (410)676-9703 

Technical  Area  Tasks:  (410)676-0200 
FAX;  (410)676-8862 
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Coastal  Engineering  Information  Analysis  Center  (CEIAC): 

Subject  coverage  -  Coastal  engineering,  coastal  regions,  beaches,  shore  erosion,  coastal 
environments,  oceanography,  ocean  waves  tides,  inlets  and  hydrodynamics.  The  POC  is: 

Commander  and  Director 

U.S.  Army  Engineer  Waterways  Experimentation  Station 
ATTN:  CEWES-CV-I 
3909  Halls  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone:  (601)634-2012 
FAX;  (601)634-3433 


Cold  Regions  Science  and  Technology  Information  Analysis  Center  (CRSTIAC) 

Subject  coverage  -  Hydrology,  climatology,  civil  engineering,  meteorology,  military  opera¬ 
tions,  mobility,  construction,  materials  in  cold  temperatures  and  environmental  issues  re¬ 
lating  to  cold  regions.  The  POC  is: 

U.S.  Army  Cold  Regions  Research  and  Engineering  Laboratory 

72  Lyme  Road 
Hanover,  NH  03755-1290 
Phone:  (603)646-4221 
FAX:  (603)646-4712 


Concrete  Technology  Information  Analysis  Center  (CTIAC) 

Subject  coverage  -  Concrete,  reinforced  concrete,  reinforcing  materials,  cements,  mixtures, 
construction  materials,  loads  (force),  fracture  (mechanics),  deformation,  degradation,  chemi¬ 
cal  analysis,  repair,  evaluation,  maintenance  and  rehabilitation.  The  POC  is: 

Commander  and  Director 

U.S.  Army  Engineer  Waterways  Experimentation  Station 
ATTN:  CEWES/SV-Z 
3909  Halls  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone:  (601)634-3264 
FAX:  (601)634-3242 
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Crew  Station  Ergonomics  Information  Analysis  Center  (CSERIAC) 

Subject  coverage  -  Scientific  and  technical  knowledge  and  data  concerning  human  charac¬ 
teristics,  abilities,  limitations,  physiological  needs  and  tolerances,  performance,  body  di¬ 
mensions,  biomechanical  dynamics  and  physical  strength.  Also  includes  engineering  and 
design  data  concerning  equipment  intended  to  be  used,  operated,  maintained  or  controlled 
in  sea,  land,  air  and  space  environments.  The  POC  is; 

CSERIAC  Program  Office 
AUCFH/CSERIAC,  Bldg  248 
2255  H  Street 

Wright-Patterson  AFB,  OH  45433-7022 
Phone;  (513)255-4842  or  DSN  785-4842 
FAX;  (513)255-4823  or  DSN  785-4823 


Data  and  Analysis  Center  for  Software  (DACS) 

Subject  coverage  -  Supports  the  development,  testing,  validation,  transitioning  of  software 
engineering  technology  to  the  defense  community,  industry  and  academia.  Includes  the 
entire  software  life  cycle  (requirements  definition,  design,  coding,  integration,  testing  and 
post-deployment  support).  The  POC  is; 

Data  &  Analysis  Center  for  Software 
Kaman  Sciences  Corporation 
P.O.  Box  120 
Utica,  NY  13503-0120 
Phone;  (315)734-3696 
FAX;  (315)734-3699 


DoD  Nuclear  Information  Analysis  Center  (DASIAC) 

Subject  coverage  -  Nuclear  weapons  explosion  phenomena,  effects  on  military  strategic 
and  tactical  systems  and  components;  survivability,  vulnerability  and  hardening,  military 
doctrine  and  operations,  nuclear  weapon  effects  testing.  The  POC  is; 

DASIAC 

2560  Huntington  Avenue 
Suite  500 

Alexandria,  VA  22303-1490 
Phone;  (703)960-4774 
FAX;  (703)329-7198 
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Guidance  and  Control  Information  .Analysis  Center  (GACIAC) 

Subject  coverage  -  Dissemination  and  exchange  of  technical  information  related  to  the 
guidance  and  control  of  weapons.  These  include  missiles,  rockets,  bombs,  submunitions, 
projectiles,  mines  and  munition  dispensing  canisters.  The  POC  is; 

I  IT  Research  Institute 
GACIAC 

10  West  35th  Street 
Chicago,  IL  60616-3799 
Phone:  (312)567-4345/4492 
FAX:  (312)567-4889 

High  Temperature  Materials  Information  Analysis  Center  (HTMIAC) 

Subject  coverage  -  Central  source  of  engineering  data  and  technical  information  on  high 
temperature  materials,  properties  and  laser  effects;  especially  in  the  critical  technology 
areas  of  aerospace  structural  composites  and  metals,  infrared  detector  materials  and  coat¬ 
ings.  The  POC  is; 

HTMIAC/CINDAS 
Purdue  University 
2595  Yeager  Road 
West  Lafayette,  IN  47906-1398 
Phone;  (317)494-9393 
FAX;  (317)496-1175 

Hydraulic  Engineering  Information  Analysis  Center  (HEIAC) 

Subject  coverage  -  River,  harbor  and  tidal  hydraulics;  flow  through  pipes,  conduits,  chan¬ 
nels  and  spillways  as  related  to  flood  control  and  navigation;  hydraulic  design  and  perfor¬ 
mance  of  dams,  locks,  channels  and  other  structures.  The  POC  is: 

Commander  and  Director 

U.S.  Army  Engineer  Waterways  Experimentation  Station 
ATTN:  CEWES/HV-Z 
3909  Hails  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone;  (601)634-2608 
FAX;  (601)634-4158 


Infrared  Information  Analysis  Center  (IRIA) 


Subject  coverage  -  Electro-optics  technology  of  interest  to  DoD  including;  sources  of  elec¬ 
tromagnetic  radiation  from  the  ultraviolet  through  far  infrared  spectral  regions;  radiation 
characteristics  of  natural  and  man-made  targets;  optical  properties  of  materials;  detection 
materials  and  elements;  information  processing  as  it  pertains  to  sensory  collection  of  data; 
imaging,  detecting,  searching,  homing,  tracking  and  ranging  subsystems.  The  POC  is; 

Environmental  Research  Institute  of  Michigan 
ATTN;  The  IRIA  Center 
P.O.  Box  134001 
Ann  Arbor,  MI  48113-4001 
Phone;  (313)994-1200  ext.  2302 
FAX;  (313)994-5550 


Manufacturing  Technology  Information  Analysis  Center  (MTIAC) 

Subject  coverage  -  Collection,  analysis  and  dissemination  of  manufacturing  technology 
and  data.  Technology  information  is  acquired  in  the  following  defense-related  fields;  ma 
chine  tools  and  manufacturing  equipment,  robots  and  special  machines,  material  handling 
equipment,  controls,  software  and  databases,  communication  lines  and  network*,  sensors, 
inspection  or  checkout  procedures,  factoiy  automation,  computer-integrated  manufactur¬ 
ing,  specific  defense-related  products,  and  the  management  aspects  of  manufacturing  tech¬ 
nology.  The  POC  is; 


MTIAC 

IIT  Research  Institute 
10  West  35th  Street 
Chicago,  IL  60616-3799 
Phone;  (312)567-4732 
FAX;  (312)567-4736 


Metals  Information  Analysis  Center  (MIAC) 

Subject  coverage  -  Central  source  of  engineering  and  technical  data  and  research  and  de¬ 
velopment  information  on  monolithic  metals,  metal  alloys,  intermetallic  compounds  and 
coatings  used  in  defense  systems.  The  POC  is; 

MIAC/CINDAS 
Purdue  University 
2595  Yeager  Road 
We.st  Lafayette,  IN  47906-1398 
Phone;  (317)494-9393 
FAX;  (317)496-1175 
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Metal  Matrix  Composites  Information  Analysis  Center  (MMCIAC) 

Subject  coverage  -  Central  source  of  engineering  and  technical  data,  and  research  and 
development  information  on  metal  matrix  composites  (MMC).  Information  pertaining  to 
aspects  of  MMC  applications  in  air,  sea,  land  and  space-based  systems  necessary  for  sup¬ 
port  of  DoD  basic  and  applied  research.  The  POC  is; 

MMCIAC/CINDAS 
Purdue  University 
2595  Yeager  Road 
West  Lafayette,  IN  47906-1398 
Phone;  (317)494-9393 
FAX;  (317)496-1175 


Nondestructive  Testing  Information  Analysis  Center  (NTIAC) 

Subject  coverage  -  Information  pertaining  to  non-destructive  testing,  non-destructive  evalu¬ 
ation,  and  non-destructive  inspection.  Coverage  includes  techniques  and  processes.  The 
use  of  nondestructive  sensors  for  manufacturing  and  materials  process  control,  and  for 
intelligent  or  adaptive  control  applications,  is  also  within  the  purview  of  NTIAC.  The 
POC  is; 


Texas  Research  Institute  Austin 
415A  Crystal  Creek  Drive 
Austin,  TX  78746-6201 
Phone;  (512)263-2106 
FAX;  (512)263-3530 


Plastics  Technical  Evaluation  Center  (PLASTEC) 

Subject  coverage  -  Acquisition,  evaluation  and  exchange  of  technical  information  related 
to  plastics,  adhesives,  and  organic  matrix  composites.  Coverage  includes  technology  from 
applied  research  through  fabrication,  with  emphasis  on  properties  and  performance.  The 
POC  is; 


Plastics  Technical  Evaluation  Center  (PLASTEC) 

U.S.  Army  Armament,  Munitions  and  Chemicals  Command 
Picatinny  Arsenal,  NJ  07806-5000 
Phone;  (201)724-4222/5859 
FAX;  (201)361-7378 
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Reliability  Analysis  Center  (RAC) 


Subject  coverage  -  Reliability,  maintainability  and  quality  of  devices  and  systems.  Reliabil¬ 
ity  and  failure  mode/mechanism  information  that  is  generated  during  all  phases  of  compo¬ 
nent  fabrication,  testing,  equipment  assembly  and  operation.  Information  and  data  in¬ 
clude  process  control,  quality  assurance  practices,  screening  and  burn-in,  qualification  and 
environmental  screening,  failure  analysis,  reliability  prediction  and  demonstrations,  field 
testing  and  mission  deployment.  Reliability  effects  of  electrical  overstress  and  electrostatic 
discharge  (EOS/ESD)  on  semiconductors  is  a  specialty.  The  POC  is: 

Reliability  Analysis  Center  (RAC) 

IIT  Research  Institute 
P.O.  Box  4700 
Rome,  NY  13440-8200 
Phone:  (315)337-0900 
FAX:  (315)337-9932 

Soil  Mechanics  Information  and  Analysis  Center  (SMIAC) 

Subject  coverage  -  Soil  mechanics,  engineering  geology,  rock  mechanics,  soil  dynamics, 
earthquake  engineering,  earth  and  rockfill  dams,  levees,  earth  retaining  structures  and 
building  foundations,  and  laboratory  testing  of  soils  and  rocks.  The  POC  is: 

Commander  and  Director,  U.S.  Army  Engineer  Waterways  Experimentation  Station 

ATTN:  CEWES/GV-Z 
3909  Halls  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone:  (601)634-3376 
FAX:  (601)634-3139 

Survivability/Vulnerability  Information  and  Analysis  Center  (SURVIAC) 

Subject  coverage  -  The  DoD  focal  point  for  non-nuclear  survivability/vulnerability  data, 
information,  methodologies,  models  and  analyses  relating  to  U.S.  and  foreign  aeronautical 
and  surface  systems.  Information  and  data  covers  the  survivability  of  all  allied  and  other 
non-adversary  systems  to  threat  weapons  as  well  as  the  effectiveness  of  U.S.  weapons  against 
foreign  systems.  The  POC  is: 


SURVIAC 

Booz-Allen  &  Hamilton 
WL/FIVS/SURVIAC 
2130  Eighth  St.,  Bldg  45,  Suite  1 
Wright-Patterson  AFB,  OH  45433-7542 
Phone:  (513)255-4840/9509  or  DSN  785-4840 
FAX:  (513)255-9673  or  DSN  785-9673 
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Tactical  Warfare  Simulation  and  Technology  Information  and  Analysis  Center  (TWSTIAC) 

Subject  coverage  -  Principal  focus  is  to  meet  operational  requirements  and  the  underlying 
acquisition  and  training  needs  associated  with  non-nuclear  roles  and  missions  for  all  tacti¬ 
cal  forces  of  the  DoD.  The  TWSTIAC  is  the  DoD  resource  for  assistance  in  developing  and 
implementing  M&S  to  operate  in  the  DoD  DIS  environment.  It  is  specifically  mandated  to 
perform  primary  research;  develop  and  apply  advanced  and  integrated  M&S;  develop  stan¬ 
dardized  processes  to  collect,  analyze,  evaluate,  and  synthesize  test  data.  The  POC  is: 

Battelle  Memorial  Institute 
ATTN:  TWSTIAC 
505  King  Avenue 
Columbus,  OH  43201-2693 
Phone:  (614)424-5047 
FAX:  (614)424-4874 

Tactical  Warfare  Technology:  (614)424-7871 
Distributed  Interactive  Simulation  (DIS)  and  related 
M&S  information  requirements:  (407)658-5014 


MILITARY  SERVICE-SPONSORED  INFORMATION  CENTERS 

Several  DoD  components  have  established  information  centers  which  provide  products 
and  services  comparable  to  those  provided  by  the  DoD  lACs.  Points  of  contact  for  these 
centers  are  provided  below.  Users  should  contact  these  organizations  directly  for  further 
information. 


U.S.  Air  Force  Aerospace  Structures  Information  Analysis  Center  (ASIAC); 

Subject  Coverage  -  A  central  agency  for  the  collection  and  dissemination  of  information 
on  aerospace  structures.  It  provides  state-of-the-art  solutions  to  small  complex  structure 
problems  and  distributes  structural  computer  programs  not  available  at  other  dissemina¬ 
tion  centers.  The  POC  is: 

Aerospace  Structures  Information  and  Analysis  Center 
ATTN:  WUFIBRA/ASIAC 
Wright-Patterson  AFB,  OH  45433-6553 
Phone:(513)255-6688  or  DSN:  785-6688 
FAX:  (513)  476-4682 
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U.S.  Navy  Shock  and  Vibration  Information  Analysis  Center  (SAVIAC): 


Subject  Coverage  -  Research,  analysis  and  testing  related  to  the  structural  dynamics,  me¬ 
chanics  and  physical  environmental  effects  on  vehicles,  structures,  equipment,  components 
and  humans  under  operational  and  combat  conditions.  This  encompasses  technical  areas 
of  vibration,  shock,  blast,  crash,  impact,  penetration,  vibroacoustics  and  mechanical  envi¬ 
ronments  along  with  supporting  areas  of  software,  sensors,  instrumentation  and  dynamic 
material  properties.  The  SAVIAC  operates  under  the  direction  of  a  multiple-agency  Tech¬ 
nical  Advisory  Group  with  members  from  the  Army,  Navy,  Air  Force,  DNA,  NASA  and 
DOE  laboratories.  The  POC  is: 

Shock  and  Vibration  Information  Analysis  Center 
2711  Jefferson  Davis  Highway  #600 
Arlington,  VA  22202-4158 
Phone:  (703)  412-7570 
FAX:  (703)  412-6555 


U.S.  Air  Force  Supportability  Investment  Decision  Analysis  Center  (SIDAC): 

Subject  Coverage:  Acquire,  improve  and  apply  existing  analysis  methods,  models,  tech¬ 
niques  and  enabling  services  for  all  aspects  of  weapon  system  supportability.  It  focuses  in 
the  areas  of  logistics  support,  logistics  research  and  development,  technology  insertion  and 
supportability  investment  information.  The  POC  is: 

Supportability  Investment  Decision  Analysis  Center  (SIDAC) 

ATTN:  SIDAC  Program  Director 
Battelle  Technical  Support  Operations,  Dayton 
5100  Springfield  Pike,  Suite  311 
Dayton,  OH  45431 

Phone:  (513)  258-6711  or  1-800-547-4322 
FAX:  (513)  254-9575 
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APPENDIX  G 

MODELING  AND  SIMULATION  TEMPLATES  FOR  USE  IN 
WEAPON  SYSTEMS  ACQUISITION  ACTIVITIES 


This  appendix  contains  sample  templates  for  the  application  of  modeling  and  simulation 
(M&S)  to  some  of  the  activities  described  in  Chapter  5.  These  templates  serve  only  as  a 
general  guideline  to  help  acquisition  managers  in  planning  how  they  will  use  models  and 
simulations  in  support  of  their  programs.  They  are  meant  to  stimulate  ideas  for  planning 
and  not  to  be  directly  applied  to  a  particular  program.  The  activities  and  M&S  tools  used 
will  vary  from  one  program  to  another.  Therefore,  such  templates  must  be  tailored  specifi¬ 
cally  for  each  individual  acquisition  program  including  the  specific  models  and  simulations 
planned  to  support  each  activity  or  document  preparation  within  the  program.  The  func¬ 
tional  activities  addressed  within  this  appendix  include;  requirements  definition;  program 
management;  design  and  engineering;  manufacturing;  test  and  evaluation;  training;  and 
logistics  support. 

In  these  templates,  readers  will  notice  that  models  and  simulations  span  across  multiple 
phases  of  a  program  and  that  the  same  type  of  models  and  simulations  are  often  used  to 
support  multiple  acquisition  activities.  Recognizing  this,  acquisition  managers  should: 

•  Plan  on  M&S  reuse  to  avoid  duplicative  M&S  development  efforts  and  to  improve 
consistency  across  the  acquisition  process; 

•  Consider  how  the  various  functional  disciplines  can  share  the  information  from  such  models 
and  simulations,  to  facilitate  Integrated  Product  and  Process  Development  (IPPD);  and 

•  Plan  for  the  transition  of  developmental  models  and  simulations  to  support  activities 
in  later  phases,  such  as  to  support  weapon  system  upgrades  or  the  training  environment. 


G-1 


Modeling  and  Simulation  Application  in  Requirements  Definition 
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Figure  G-1 .  Modeling  and  Simulation  Application  in  Requirements  Definition 


Modeling  and  Simulation  Application  in  Design  and  Engineering 
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Figure  G-3.  Modeling  and  Simulation  Application  in  Design  and  Engineering 
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Modeling  and  Simulation  Application  in  Test  and  Evaluation 


Figure  G-5.  Modeling  and  Simulation  Application  in  Test  and  Evaluation 
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APPENDIX  H 

ABBREVIATIONS  AND  ACRONYMS  USED  THROUGHOUT 
THIS  MODELING  AND  SIMULATION  GUIDEBOOK 


6-DOF 

ACAT 

ACE-IT 

ACMC 

ADM 

ADS 

AFAM 

AFMC 

AFSAA 

ALM 

AMC 

AMIP 

AMSAA 

AMSEC 

AMSMO 

AMSMP 

AOR 

APB 

APMIAC 

\R 

ARPA 

ASD(PA&E) 

ASN(RDA) 

ATD 

ATFM&S 

AWC 

AWSIM 

BAT 

BFTT 

CAA 

CAAM 

CAD 

CAE 

CAIG 

CALS 

CAM 

CARD 

CASTFOREM 


Six  Degree-of-Freedom 
Acquisition  Category 

Automated  Cost  Estimating  -  Integrated  Tools 

Assistant  Commandant  of  the  Marine  Corps 

Acquisition  Decision  Memorandum 

Advanced  Distributed  Simulation 

Air  Force  Acquisition  Model 

Air  Force  Materiel  Command 

Air  Force  Studies  and  Analysis 

Airlift  Loading  Model 

Army  Materiel  Command 

Army  Model  Improvement  Program 

U.S.  Army  Material  Systems  Analysis  Activity 

Army  Modeling  &  Simulation  Executive  Council 

Army  Model  &  Simulation  Management  Office 

Army  Model  &  Simulation  Management  Progra*. 

Area  of  Responsibility 
Acquisition  Program  Baseline 

Airfields,  Pavements,  and  Mobility  Information  Analysis  Center 
Army  Regulation 

Advanced  Research  Projects  Agency 

Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 
Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition) 

Advanced  Technology  Demonstrations 

Acquisition  Task  Force  on  Modeling  &  Simulation 

U.S.  Army  War  College 

Air  Warfare  Simulation  Model 

Brilliant  Anti-Tank 

Battle  Force  Tactical  Training 

US  Army  Concepts  Analysis  Agency 

Composite  Area  Analysis  Model 

Computer-Aided  Design 

Computer-Aided  Engineering 

Cost  Analysis  Improvement  Group 

Continuous  Acquisition  and  Life-Cycle  Support 

Computer-Aided  Manufacturing 

Cost  Analysis  Requirements  Document 

Combined  Arms  and  Support  Task  Force  Evaluation  Model 
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CATIA 

CBIAC 

CCTT 

CEAC 

CECOM 

CED 

CEIAC 

CEM  V 

CIAC 

CINC 

CJCS 

CM 

COEA 

CPIA 

CRSTIAC 

CSEAL 

CSERIAC 

CTIAC 

DA 

DAB 

DACS 

DAE 

DASIAC 

DDR&E 

DDTO 

DemVal 

DepSecDef 

DIA 

DIS 

DMS 

DMSO 

DoD 

DON 

DPG 

DSB 

DSMC 

DT 

DT&E 

DTIC 

DUSA(OR) 

EADSIM 

ECP 

EMD 

ENWGS 


Computer-Aided  Three  Dimensional  Interactive  Application 
Chemical  Warfare/Chemical  and  Biological  Defense  Information 
Analysis  Center 
Close  Combat  Tactical  Trainer 
U.S.  Army  Cost  and  Economic  Analysis  Center 
U.S.  Army  Communications-Electronics  Command 
Concept  Exploration  and  Definition 
Coastal  Engineering  Information  Analysis  Center 
Concepts  Evaluation  Model  V 
Ceramics  Information  Analysis  Center 
Commander-in-Chief 
Chairman  of  the  Joint  Chiefs  of  Staff 
Configuration  Management 
Cost  and  Operational  Effectiveness  Analysis 
Chemical  Propulsion  Information  Agency 

Cold  Regions  Science  and  Technology  Information  Analysis  Center 

Combat  System  Engineering  and  Analysis  Laboratory 

Crew  Station  Ergonomics  Information  Analysis  Center 

Concrete  Technology  Information  Center 

Department  of  the  Army 

Defense  Acquisition  Board 

Data  and  Analysis  Center  for  Software 

Defense  Acquisition  Executive 

DoD  Nuclear  Information  and  Analysis  Center 

Director  of  Defense  Research  and  Engineering 

Deputy  Director  of  Technical  Operations 

Demonstration  and  Validation 

Deputy  Secretary  of  Defense 

Defense  Intelligence  Agency 

Distributed  Interactive  Simulation 

Distributed  Models  and  Simulations 

Defense  Modeling  &  Simulation  Office 

Department  of  Defense 

Department  of  the  Navy 

Defense  Planning  Guidance 

Defense  Science  Board 

Defense  Systems  Management  College 

Developmental  Test 

Developmental  Testing  and  Evaluation 

Defense  Technical  Information  Center 

Deputy  Under  Secretary  of  the  Army  for  Operations  Research 

Extended  Air  Defense  Simulation 

Engineering  Change  Proposal 

Engineering  and  Manufacturing  Development 

Enhanced  Naval  Warfare  Gaming  System 


EOS/ESD 

ESAMS 

EXCIMS 

FMF 

FOT&E 

GACIAC 

GWEF 

HQMC 

HTMIAC 

HW/SWIL 

HWIL 

lAC 

IDA 

IG 

lOT&E 

IPPD 

IPR 

IPT 

IRIA 

IWSM 

JADS/JFS 

JM&S 

JMSEP 

JROC 

JTCG/AS 

JTCTS 

EAR 

LCC 

LFT 

LMI 

LOGSA 

LORA 

LOS-F-H 

LRIP 

LSA 

LSAR 

M&S 

M/ETC 

MAA 

MAGTF 

MAP 

MASS 

MCCDC 

MCMSMO 

MCMSWG 


Electrical  Overstress  and  Electrostatic  Discharge 
Enhanced  Surface-to-Air  Missile  Simnulation 
Executive  Council  on  Modeling  &  Simulation 
Fleet  Marine  Force 

Follow-on  Operational  Test  and  Evaluation 
Guidance  and  Control  Information  Analysis  Center 
Guided  Weapons  Evaluation  Facility 
Headquarters  Marine  Corps 

High  Temperature  Materials  Information  Analysis  Center 

hardware/software-in-the-loop 

hardware-in-the-loop 

Information  Analysis  Center 

Institute  for  Defense  Analysis 

Inspector  General 

Initial  Operational  Test  and  Evaluation 

Integrated  Product  and  Process  Development 

In  Process  Review 

Integrated  Product  Team 

Infrared  Information  Analysis  Center 

Integrated  Weapon  System  Management 

Joint  Advanced  Distributed  Simulation  Joint  Feasibility  Study 

Joint  Models  and  Simulations 

Joint  Modeling  and  Simulation  Executive  Panel 

Joint  Requirements  Oversight  Council 

Joint  Technical  Coordinating  Group  on  Aircraft  Survivability 

Joint  Tactical  Combat  Training  System 

Launch  Acceptability  Regions 

Life-Cycle  Cost 

Live  Fire  Testing 

Logistics  Management  Institute 

U.S.  Army  Materiel  Command  Logistics  Support  Agency 

Level  of  Repair  Analysis 

Line-of-Sight  Forward,  Heavy 

Low  Rate  Initial  Production 

Logistics  Support  Analysis 

Logistics  Support  Analysis  Records 

Modeling  and  Simulation 

MAGTF/Expeditionary  Training  Center 

Mission  Area  Assessment 

Marine  Air-Ground  Task  Force 

Mission  Area  Plan 

Mobility  Analysis  Support  System 

Marine  Corps  Combat  Development  Command 

Marine  Corps  Modeling  «&  Simulation  Management  Office 

Marine  Corps  Modeling  &  Simulation  Working  Group 
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MDA 

MDAP 

MIAC 

MICOM 

MMC 

MMCIAC 

MNA 

MNS 

MOE 

MOO 

MOP 

MOP  77 

MOSAIC 

MOU 

MRS-BURU 

MS 

MS&A 

MSIS 

MTIAC 

MTWS 

NAVAIR 

NDAA 

NDI 

NLOS 

NTIAC 

O&S 

OAS 

OFP 

OPTEC 

ORD 

OSD 

OT 

OT&E 

PA&E 

PDM 

PE 

PEO 

PLASTEC 

PM 

PMD 

PMOs 

PMWS 

POC 

POM 


Milestone  Decision  Authority 
Major  Defense  Acquisition  Program 
Metals  Information  Analysis  Center 
U.S.  Army  Missile  Command 
Metal  Matrix  Composites 

Metals  Matrix  Composites  Information  Analysis  Center 

Mission  Need  Analysis 

Mission  Need  Statement 

Measures  of  Effectiveness 

Measures  of  Outcome 

Measures  of  Performance 

Memorandum  of  Policy  No.  77 

Models  &  Simulations:  Army  Integrated  Catalog 

Memorandum  of  Understanding 

Mobility  Requirements  Study-Bottom -Up  Review  Update 
Milestone  (as  in  MS  0,  MS  I,  etc.) 

Modeling  &  Simulation  and  Analysis 
Modeling  and  Simulation  Information  System 
Manufacturing  Technology  Information  Analysis  Center 
Marine  Air-Ground  Task  Force  (MAGTF)  Tactical  Warfare 
Simulation 

Naval  Air  Systems  Command 
Non-Developmental  Airlift  Aircraft 
Non-Developmental  Item 
Non  Line-of-Sight 

Nondestructive  Testing  Information  Analysis  Center 
Operations  and  Support 

Office  of  Aerospace  Studies  (Air  Force  Materiel  Command) 
Operational  Flight  Program 

US  Army  Operational  Test  and  Evaluation  Command 

Operational  Requirements  Document 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Operational  Test  and  Evaluation 

Program  Analysis  and  Evaluation 

Periodic  Depot  Maintenance 

Program  Element 

Program  Executive  Office 

Plastics  Technical  Evaluation  Center 

Program  Manager 

Program  for  Management  Development 
Program  Management  Offices 
Program  Manager’s  Work  Station 
Point  of  Contact 

Program  Objective  Memorandum 
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PPBS 

PRIMES 

R&D 

RAC 

RAM 

RCM 

RD&A 

RDEC 

RFP 

S&A 

SAB 

SAFMA 

SAIC 

SEMP 

SIDAC 

SIMNET 

SIMTECH 

SIMWG 

SMIAC 

SOF 

SOW 

SPAWAR 

SSP 

SSTORM 

STAF 

STRICOM 

SURVIAC 

SYSCOM 

T&E 

T&E 

TACOM 

TARDEC 

TECOM 

TEMP 

TLD 

TPFDD 

TPIPT 

TRAC 

TRADOC 

IRIMS 

TWSTIAC 

USAF/CVA 


Planning,  Programming  &  Budgeting  System 

Preflight  Integration  of  Munitions  and  Electronic  Systems 

Research  and  Development 

Reliability  Analysis  Center 

Reliability,  Availability,  and  Maintainability 

Requirements  Correlation  Matrix 

Research,  Development  and  Acquisition 

Research.  Development  and  Engineering  Center 

Request  for  Proposal 

Studies  and  Analysis 

Air  Force  Scientific  Advisory  Board 

Strategic  Airlift  Force  Mix 

Science  Applications  International  Corporation 

Systems  Engineering  Management  Plan 

Supportability  Investment  Decision  Analysis  Center 

Simulation  Network 

Simulation  Technology  Program 

Simulation  Working  Group 

Soil  Mechanics  Information  and  Analysis  Center 

Special  Operation  Forces 

Statement  of  Work 

Space  and  Naval  Warfare  Systems  Command 
Simulation  Support  Plan 

Structured  Scenario  Torpedo  Operational  Requirements  Model 

Simulation/Test  Acceptance  Facility 

Simulation,  Training  and  Instrumentation  Command 

SurvivabilityA^ulnerability  Information  Analysis  Center 

Systems  Command 

test  and  evaluation 

Training  and  Education  Division,  Marine  Corps  Combat 
Development  Command 
U.S.  Army  Tank-Automotive  Command 
U.S.  Army  Tank  Automotive  and  Armament  Research, 
Development  &  Engineering  Ctr 
U.S.  Army  Test  and  Evaluation  Command 
Test  and  Evaluation  Master  Plan 
Top  Level  Demonstrations 
Time-Phased  Force  and  Deployment  Data 
Technical  Planning  Integrated  Product  Team 
TRADOC  Analysis  Command 
U.S.  Army  Training  and  Doctrine  Command 
Technical  Risk  Identification  and  Mitigation  System 
Tactical  Warfare  Simulation  and  Technology  Information  and 
Analysis  Center 

Assistant  Vice  Chief  of  Staff,  U.S.  Air  Force 
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USAF/XOM 

USD(A&T) 

USD(AR) 

v«&v 

VV&A 

WBS 

WEPTAC 


U.S.  Air  Force  Directorate  of  Modeling  ,  Simulation  and  Analysis 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

Under  Secretary  of  Defense  for  Acquisition  Reform 

Verification  and  Validation 

Verification,  Validation  and  Accreditation 

Work  Breakdown  Structure 

Weapons  and  Tactics  Center 
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APPENDIX  I 
GLOSSARY 


Acceptability  criteria:  A  set  of  standards  that  a  particular  M&S  must  meet  to  be  accredited 
for  a  given  use.  (DA  PAM  5-11) 

Accreditation:  The  official  certification  that  a  model  or  simulation  is  acceptable  for  use  for 
a  specific  purpose.  (DoDD  5000.59) 

Ada:  A  high  order  computer  language  designed  and  developed  to  DoD  requirements  for 
modular  standard  language.  While  the  original  focus  was  for  real-time  embedded  software, 
Ada  has  also  been  used  for  a  variety  of  other  software  systems  including  some  simulation 
systems. 

Advanced  Distributed  Simulation  (ADS):  A  concept  which  applies  a  common  core  of  ad¬ 
vanced  technologies  (including;  computer,  display,  communication  and  simulation)  to  pro¬ 
vide  a  mix  of  live,  constructive  and  virtual  simulation  methods  across  the  spectrum  of  De¬ 
fense  uses,  from  training  and  readiness  through  requirements  generation  through 
prototyping  through  fielding.  ADS  and  DIS  are  synonymous.  (DSB) 

Aggregate  Level  Simulation  Protocol  (ALSP):  A  protocol  that  permits  the  integration  of 
distributed  simulations  (or  Actors).  The  protocol  is  currently  being  used  to  integrate  AWSIM 
and  CBS.  The  protocol  synchronizes  the  advancement  of  simulation  time  among  the  simu¬ 
lations,  provides  mechanisms  for  interaction  among  combat  entities  (e.g.,  direct  or  indirect 
fire  engagements)  across  simulations,  and  the  update  of  state  attributes  of  those  combat 
entities. 

Air  Warfare  Simulation  Model  (AWSIM):  The  simulation  currently  used  at  the  Warrior 
Preparation  Center  and  Blue  Flag  to  conduct  battle  staff  training. 

Algorithm:  A  prescribed  set  of  well-defined,  unambiguous  rules  or  processes  for  the  solu¬ 
tion  of  a  problem  in  a  finite  number  of  steps.  (Webster  Computer) 

Appended  TVainers:  Combat  Vehicle  Appended  Trainers  are  a  family  of  deployable  train¬ 
ers  designed  to  support  individual  and  full-crew  mission  training  for  tanks,  armored  ve¬ 
hicles  and  assault  vehicles.  These  trainers  include  the  visual  and  aural  cues  which  immerse 
the  operator  in  a  near-real  battle  environment,  and  have  provisions  for  recording  events 
for  post-exercise  crew  performance  analysis. 

Architecture:  The  high-level  organization  of  hardware  or  software  systems.  (Krueger) 

Artificial  Intelligence:  The  effort  to  automate  those  human  skills  that  illustrate  our  intelli¬ 
gence  (e.g.,  understanding  visual  images,  understanding  speech  and  written  text,  problem 
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solving  and  medical  diagnosis).  (Krueger) 


Brawler:  TAC  Brawler  is  an  engagement  level  air-to-air  combat  simulation  tool  used  pri¬ 
marily  for  the  evaluation  of  avionics,  weapons  and  tactics.  It  has  also  been  used  as  a  target 
generator  for  manned  simulators.  (J-8  Catalog). 

C-Plus-Plus  (C-f  +  ):  A  high  order  computer  language  used  extensively  in  commercial  soft¬ 
ware.  C-H-  is  an  object  oriented  extension  to  the  C  language. 

Classes  of  Simulation:  (DSB) 

Live  -  The  live  component  of  simulation  involves  operations  with  real  forces  and  real 
equipment  in  the  air,  on  the  ground,  on  and  below  the  sea. 

Constructive  -  A  class  of  simulation  typified  by  wargames,  models  and  analytical  tools. 

Virtual  -  A  class  of  simulation  where  systems  are  simulated  both  physically  and  elec¬ 
tronically. 

Code  Verification:  A  rigorous  audit  of  all  compilable  code  to  ensure  that  the  representa¬ 
tions  of  vertical  logic  have  been  properly  implemented  in  the  computer  code.  (DA  PAM  5- 
11) 

Combined  Arms  and  Support  Task  force  Evaluation  Model  (CASTFOREM):  This  is  the 
Army’s  highest  resolution,  lowest  echelon  (up  through  and  including  brigade)  combined 
arms  combat  simulation  model.  Often  run  in  conjunction  with  JANUS(T)  to  perform  weapon 
systems  analysis  for  a  COEA  (MOSAIC). 

Commcn-use  M&S:  M&S  applications,  services  or  materials  provided  by  a  DoD  compo¬ 
nent  to  two  or  more  DoD  components.  (DoDD  5000.59) 

Compatibility:  The  capability  of  a  functional  unit  to  meet  the  requirements  of  a  specified 
interface.  (ANSI  X3.172-1990) 

Computer  Generated  Forces  (CGF):  A  collection  of  unmanned  battlefield  entities  under 
control  as  a  unit.  CGF  replace  or  supplement  friendly,  enemy  or  neutral  manned  simula¬ 
tors  during  a  specific  session.  The  SIMNET  program  uses  the  term  “semi-automated  forces” 
(SAFOR)  for  CGF.  (DIS  Glossary) 

Computer  War  Game:  A  technique  by  which  different  concepts,  different  pieces  of  hard¬ 
ware  or  different  military  plans  can  be  investigated  in  a  multi-sided  confrontation  using  a 
computer  to  generate  displays  of  the  battlefield  and  perform  computations  of  outcomes. 

Concepts  Evaluation  Model  (CEM):  CEM  is  used  to  analyze  force  effectiveness  at 
theater  level  warfare.  It  is  used  as  a  tool  to  assess  the  effectiveness  of  different  mixes 
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of  forces  or  resources  and  estimates  of  ammunition,  equipment  and  personnel  require¬ 
ments.  (MOSAIC) 

Concurrent  Engineering:  Concurrent  engineering  is  a  systematic  approach  to  the  integrated, 
concurrent  design  of  products  and  their  related  processes,  including  manufacture  and  sup¬ 
port.  This  approach  is  intended  from  the  outset  to  cause  developers  to  consider  all  ele¬ 
ments  of  the  product  life  cycle  from  conception  through  disposal,  including  quality,  cost, 
schedule  and  user  requirements.  (IDA) 

Confederation  of  Models:  A  set  of  simulation  methods  (or  Actors)  operating  in  an  inte¬ 
grated  manner  using  the  ALSP  protocols. 

Continuous  System:  A  system  for  which  the  state  variables  change  continuously  with  re¬ 
spect  to  time.  (Law'  and  Kelton) 

Cooperative  Development:  A  project  in  which  two  or  more  DoD  components  share  in  do¬ 
main  research,  technical  studies  or  technology  development,  but  that  may  result  in  dissimi¬ 
lar  M&S  applications.  (DoDD  5000.59) 

Corps  Battle  Simulation  (CBS):  A  simulation  used  by  the  Army  simulation  centers  to  train 
battle  staffs  at  Corps  and  echelons  below.  Previously  known  as  the  Joint  Exercise  Simula¬ 
tion  System  (JESS). 

Cross-functional  Integration:  The  melding  of  acquisition  functions  (such  as  design  analysis 
with  logistics  analysis)  involving  shared  modeling  and  simulations  data  and  information. 
(ATFM&S) 

Data  Base  Management  System:  A  set  of  computer  programs  that  provides  convenient  and 
efficient  means  to  retrieve  and  store  data  in  a  database. 

Data  Base:  A  collection  of  data. 

Defense  Simulation  Internet  (DSI):  A  communication  network  under  development  by  ARPA 
that  provides  secure,  packet-switched,  data,  voice  and  video  services.  Current  system  pro¬ 
vides  communication  links  at  1.5  megabits  per  second  rates.  DSI  should  not  be  confused  , 
with  DIS:  DSI  may  be  used  as  a  communication  network  over  which  DIS  simulators  are 
linked.  However,  DSI  is  not  limited  to  supporting  the  linking  of  DIS  simulators,  nor  are 
DIS  simulators  limited  to  using  the  DSI  as  a  communication  network. 

Deterministic  Algorithm:  A  process  that  yields  a  unique  and  predictable  outcome  for  a 
given  set  of  inputs.  (Harris) 

Discrete  System:  A  system  for  which  the  state  variables  change  instantaneously  at  sepa¬ 
rated  points  in  time.  (Law  and  Kelton) 
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Distributed  Interactive  Simulation  (DIS):  1.  A  time  and  space  coherent  representation  of 
a  virtual  battlefield  environment,  measured  in  terms  of  the  human  perception  and  the  be¬ 
haviors  of  warfighters  interacting  in  free  play  with  other  warfighters  and/or  with  computer 
generated  forces.  DIS  prov  des  a  structure  by  which  independently  developed  systems  may 
interact  with  each  other  in  a  well  managed  and  validated  combat  simulation  environment 
during  all  phases  of  the  development  process  and  in  subsequent  training.  DIS  and  ADS  are 
synonymous.  2.  The  class  of  simulations  defined  by  the  DIS  Architecture  and  associated 
standards.  (DIS  Glossary) 

DoD  Modeling  and  Simulation  (M&S)  Executive  Agent:  A  DoD  component  to  whom  the 
USD(A&T)  has  assigned  responsibility  and  delegated  authority  for  the  development  and 
maintenance  of  a  specific  area  of  M&S  application,  including  relevant  standards  and  data¬ 
bases,  used  by  or  common  to  many  models  and  simulations.  (DoDD  5000.59) 

Eagle:  Corps/Division  level  combat  model  that  simulates  the  operations  level  of  war  and 
includes  joint  and  combined  operations.  It  is  used  for  assessments,  combat  development,  as 
an  exercise  driver  and  as  a  staff  trainer.  (MOSAIC) 

Enhanced  Naval  Warfare  Gaming  System  (ENWGS):  A  multi-platform,  multi-warfare,  real¬ 
time  wargaming  and  training  simulation  used  for  battle  staff  training  and  strategic  and 
operational  wargaming  and  planning.  (Navy  M&S  Catalog) 

Enhanced  Surface-to-Air  Missile  Simulation  (ESAMS):  Generates  one-on-one  probability'  of 
kill  for  aircraft  versus  surface-to-air  missiles.  Its  results  are  used  in  higher  level  survivabiliu 
analyses  to  evaluate  weapon  system  and  subsystems  effectiveness.  (AFSAA  Catalog) 

Entity:  An  object  by  which  the  system  can  be  defined  (i.e.,  a  component  of  the  system 
represented  in  the  model).  (McQuay) 

Executive  Council  for  Modeling  and  Simulation  (EXCIMS):  An  organization  established 
by  the  Under  Secretary  of  Defense  (Acquisition)  (USD(A))  (now  USD(A&T))  and  re¬ 
sponsible  for  providing  advice  and  assistance  on  DoD  modeling  and  simulation  issues. 
Membership  is  determined  by  the  USD(A&T)  and  is  at  the  Senior  Executive  Service,  flag 
and  general  officer  level.  Chaired  by  the  DDR&E.  (DoD  5000.59) 

Extended  Air  Defense  Simulation  (EADSIM);  EADSIM  is  used  primarily  to  analyze  ex¬ 
tended  air-defense  scenarios.  It  is  used  to  evaluate  effectiveness  and  efficiency  of  weapon 
systems  against  targets  and  to  evaluate  the  value  of  different  mixes  of  forces  or  resources. 
(J-8  Catalog) 

Fidelity:  The  degree  to  which  aspects  of  the  real  world  are  represented  in  the  M&S.  (See 
“Resolution”)(DA  PAM  5-11) 

General-use  M&S  Applications:  Specific  representations  of  the  physical  environment  ef¬ 
fects  used  by,  or  common  to,  many  models  and  simulations;  e.g.  terrain,  atmospheric  or 
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hydrographic  effects.  (DoDD  5000.59) 


Hardware/Software-in<the-Loop:  This  hybrid  simulation  includes  actual  system  or  subsystem 
hardware  and  software  in  conjunction  with  mathematical  (computer)  models  and  external 
stimuli  to  demonstrate  the  capability  to  operate  within  an  environment  simulating  actual 
operating  conditions. 

Hierarchy:  A  hierarchy  of  models  and  simulations  is  a  taxonomy  which  is  used  to  describe 
the  various  levels  of  models  and  simulations.  Assorted  taxonomies  may  be  found  in  the 
literature;  this  guidebook  describes  four  levels  of  models  and  simulations:  engineering, 
engagement,  mission/battle  and  theater/campaign. 

Hybrid  Simulation:  A  simulation  that  combines  multiple  classes  of  simulations,  such  as 
combining  computer  (constructive)  simulations  with  actual  system  hardware  and  software 
(live).  .An  example  is  a  hardware/software-in-the-loop  simulation. 

Integrated  Product  and  Process  Development  (IPPD):  IPPD  is  an  approach  to  systems  acqui¬ 
sition  which  brings  together  all  of  the  functional  disciplines  required  to  develop,  design,  test, 
produce  and  field  a  system.  This  is  essentially  the  same  as  Concurrent  Engineering. 

Integrated  Product  Team  (IPT):  Integrated  Product  Teams  are  a  means  to  achieve  concurrent 
engineering  or  IPPD.  They  are  multidisciplinary  teams  consisting  of  representatives  from  all 
disciplines  involved  in  the  system  acquisition  process,  from  requirements  development  through 
disposal.  Having  the  participation  of  all  the  appropriate  disciplines,  IPTs  are  often  empowered 
to  make  decisions  to  achieve  successful  development  of  their  particulai  pn  duct. 

Interface;  The  interconnection  between  two  pieces  of  hardware  or  software.  A  device  or 
piece  of  software  that  accomplishes  such  a  connection.  (Krueger) 

Interactive  Models:  Models  that  require  human  participation  are  sometimes  called  inter¬ 
active  or  human-in-the-lcop.  Human  participation  can  include  decision  making  within  com¬ 
puter  wargaming  models  for  tactics  development  and  battle  staff  training  as  well  as  human- 
in -the-loop  weapon  system  simulators  and  trainers. 

Interoperability:  The  capability  of  two  or  more  systems  to  exchange  and  use  information. 
(ANSI  X3. 172-1990) 

JANUS:  JANUS  is  a  multi-purpose  near-real-time  interactive  wargame  used  to  examine 
the  relationships  of  combat  and  tactical  processes.  It  is  used  for  weapon  system  perfor¬ 
mance,  test  planning,  test  augmentation,  scenario  evaluation  and  exercises.  It  can  model 
entities  down  to  the  individual  soldier  or  system.  (MOSAIC) 

Joint  M&S:  Modeling  and  simulation  representations  of  joint  and  Service  forces,  capabili¬ 
ties,  materials  and  services;  used  in  the  joint  environment  of  by  two  or  more  Services. 
(DoDD  5000.59) 
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Legacy  Model:  A  model  developed  in  the  past  which  is  still  in  use  that  was  not  implemented 
using  today's  standards  (e.g.,  software,  communication,  DIS,  ALSP.  etc. ).  St)me  legae\  m<  idels 
have  been  modified  w  ith  interfaces  to  some  of  the  current  standards  e.xtending  their  useful 
ness  and  interoperability  with  newer,  standards  based  models. 

Marine  Air  Ground  Task  Force  (MAGTF)  Tactical  Warfare  Simulation  (MTWS):  M  1  W  S 

is  a  computer-assisted  exercise  support  tool  designed  to  support  training  of  Marine  Corps 
commanders  and  their  staffs.  It  is  designed  to  be  u.sed  in  Command  Post  Lxercises  in 
which  combat  forces,  supporting  arms,  and  combat  results  are  modeled  by  the  system.  It 
will  also  be  used  in  field  e.xercises  in  which  all,  or  part,  of  the  combat  forces  are  actual 
military  units.  MTWS  provides  a  full  spectrum  of  combat  models  required  to  support  .Marine 
Corps  exercises. 

Measure  of  Effectiveness  (MOE):  A  quantitative  expression  which  compares  the  effective¬ 
ness  of  alternatives  in  meeting  an  operational  objective  or  need.  (DA  PAM  3-11) 

Measure  of  Performance  (MOP):  A  defined  metric  of  a  component  w  hich  contributes  to 
basic  system  effectiveness  as  described  by  an  MOE.  MOPs  relate  to  specific  performance 
characteristics  from  which  data  can  be  actually  collected.  (DA  PAM  5-11) 

Measures  of  Outcome  (MOO):  Metrics  that  define  how  operational  requirements  contrib¬ 
ute  to  end  results  at  higher  levels,  such  as  campaign  or  national  strategic  outcomes. 

Model:  A  physical,  mathematical  or  otherwise  logical  representation  of  a  system  entity, 
phenomenon  or  process.  (DoDD  5000.59) 

Model  and  Simulation  (M&S)  Interoperability:  The  ability  of  a  model  or  simulation  to 
provide  .services  (data  and  functionality)  to  and  accept  services  from  other  models  and 
simulations;  and  to  use  the  services  so  exchanged  to  enable  them  to  operate  effectively 
together.  (DoDD  5000.59) 

Modeling  and  Simulation  (M&S)  Investment  Plan:  A  DoD  plan,  published  under  the  au¬ 
thority  of  the  Under  Secretary  of  Defense  (Acquisition  &  Technology)  and  with  the  coordi¬ 
nation  of  the  DoD  components,  that  established  short-term  (present  to  six  years)  and  long¬ 
term  (beyond  six  years)  programs  and  funding  for  Joint  and  common  use  M&S  to  achieve 
the  specified  goals  and  objectives  outlined  in  the  DoD  M&S  Master  Plan.  (DoDD  5000.59) 

Model-Test-Model;  An  integrated  approach  to  using  models  and  simulations  in  support  of: 
pre-test  analysis  and  planning;  conducting  the  actual  test  and  collecting  data;  and  post-test 
analysis  of  test  results  along  with  further  validation  of  the  models  using  the  test  data. 

Monte-Carlo  Algorithm:  A  statistical  procedure  that  determines  the  occurrence  of  proba¬ 
bilistic  events  or  values  of  probabilistic  variables  for  deterministic  models,  i.e.,  make  a  ran¬ 
dom  draw.  (Neelamkavil) 
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Object-Oriented  l>anguage:  A  language  which  best  suits  an  object-oriented  decomposition 
ot  software  and  which  provides  the  capability  to  implement  classes  and  objects.  Directly 
supports  data  abstraction  and  classes,  and  provides  additional  support  for  inheritance  as  a 
means  of  e.xpre.ssing  hierarchies  of  classes.  (Booch) 

Object-Oriented  Programming:  A  method  of  implementation  in  which  programs  are  orga¬ 
nized  as  cooperative  collections  of  objects,  each  of  which  represents  an  instance  of  some 
class,  and  whose  classes  are  all  members  of  a  hierarchy  of  classes  united  via  inheritance 
relationships.  (Booch) 

Object:  Physical  or  logical  structures  (mtrdels)  that  keep  their  characteristics  and  behavior 
together. 

Open  System:  A  system  in  which  the  components  and  their  composition  are  specified  in  a 
non-proprietarv  environment,  enabling  competing  organizations  to  use  these  standard  com¬ 
ponents  to  build  competitive  systems.  There  are  three  perspectives  on  open  systems;  port- 
ability-the  degree  to  which  a  system  component  can  be  used  in  various  environments, 
interoperability-the  ability  of  individual  components  to  exchange  information,  and  integra- 
tion-the  consistency  of  the  various  human-machine  interfaces  between  an  individual  and 
all  hardware  and  software  in  the  system.  (Nutt) 

Parallel  Processing:  Multiple  processes  running  on  multiple  processors  simultaneously. 

Protocol:  A  set  of  rules  used  to  control/regulate  the  interaction  between  entities  in  a  system 
(e.g.,  computers  communicating  on  a  network).  Often  implemented  as  hierarchy  of  “lay¬ 
ers”  in  which  each  level  provides  a  defined  set  of  services  to  the  layer  above. 

Real-Time  System:  A  system  that  computes  its  results  as  quickly  as  they  are  needed  by  a 
real-world  system.  Such  a  system  responds  quickly  enough  that  there  is  no  perceptible  de¬ 
lay  to  the  human  observer.  In  general  use,  the  term  is  often  perverted  to  mean  within  the 
patience  and  tolerance  of  a  human  user.  (Krueger) 

Resolution:  The  degree  of  detail  and  precision  used  in  representation  of  real  world  aspects 
in  the  M&S.  (See  “Fidelity)  (DA  PAM  5-11) 

Requirements  Correlation  Matrix  (RCM):  A  prioritized  list  of  system  requirements  with 
associated  performance  thresholds  and  goals. 

Semi-Automated  Forces  (SAFOR):  Simulation  of  friendly,  enemy  and  neutral  platforms  on 
the  virtual  battlefield  in  which  the  individual  platform  simulation  are  operated  by  com¬ 
puter  simulation  of  the  platform  crew  and  command  hierarchy.  Th term  “semi-automated” 
implies  that  the  automation  is  controlled  and  monitored  by  a  human  who  injects  command- 
level  decision  making  into  the  automated  command  process.  For  the  purpose  of  the  DIS 
architecture,  the  term  Computer  Generated  Forces  (CGF)  replaces  SAFOR.  (DIS  Glossary) 
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Scenario:  The  entire  spectrum  of  environmental  considerations  that  have  interaction  v/ith 
systems(s)  under  analysis  or  those  of  interest  for  training  purposes.  The  spectrum  includes 
physical  environment,  threat  conditions,  rules  of  engagements,  and  systems  performance 
and  effectiveness. 

SIM  II:  SIM  II  is  an  event-driven  Monte-Carlo  platform  level  engagement  simulation  for 
submarines  and  surface  ships.  (Navy  M&S  Catalog) 

SIMNET:  ARPA/Army  simulation  network  where  simulators  can  be  connected  over  local 
and  wide-area  networks  to  create  a  simulated  battlefield.  SIMNET  was  the  forerunner  to 
the  DIS  architecture. 

Simulation:  A  method  for  implementing  a  model  over  time.  Also  a  technique  for  testing, 
analysis  or  training  in  which  real-world  systems  are  u.sed,  or  where  real-world  and  concep¬ 
tual  systems  are  reproduced  by  a  model.  (DoDD  5000.59) 

State  Variables:  The  collection  of  variables  necessary  to  describe  a  system  at  a  particular 
time,  relative  to  the  objectives  of  the  study.  (Law  and  Kelton) 

Stimulation:  Stimulation  is  the  use  of  simulations  to  provide  an  external  stimulus  to  a  sys¬ 
tem  or  subsystem.  An  example  is  the  use  of  a  simulation  representing  the  radar  return  from 
a  target  to  drive  (stimulate)  the  radar  of  a  missile  system  within  a  hardware/software-in- 
the-loop  simulation. 

Stochastic  Process:  Any  process  dealing  with  events  that  develop  in  time  or  cannot  be 
described  precisely,  except  in  terms  of  probability  theory.  (Webster  Computer) 

Stochastic:  Probabili.stic  or  random,  as  oppo.sed  to  deterministic. 

Structured  Scenario  Torpedo  Operational  Requirements  Model  (SSTORM):  A  high  fidel¬ 
ity  Monte-Carol  torpedo  simulation  capable  of  modeling  the  performance  of  U.  S.  and 
threat  weapons.  It  includes  submarine  and  torpedo  kinematics  as  well  as  countermeasures. 
(Navy  M&S  Catalog) 

Synthetic  Environments:  Representations  of  present  or  future,  factory-to-battlefield,  envi¬ 
ronments  generated  by  models,  simulations  and  wargames.  May  include  a  mix  of  real  and 
simulated  objects  accessible  from  widely  dispersed  locations.  One  of  the  Science  and  Tech¬ 
nology  Thrust  areas. 

Thunder:  Thunder  is  a  two-sided  theater  level  model  designed  to  simulate  conventional, 
air-land  combat.  It  determines  contribution  of  weapon  systems  to  the  combat  outcomes  for 
key  operational  objectives.  (AFSAA  Catalog) 

Virtual  Prototype:  A  computer-based  simulation  of  systems  and  subsystems  which  exhibits 
both  geometric  and  functional  realism.  This  three-dimensional  virtual  mockup  may  be  used 
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to  evaluate  prototypes  or  concepts  and  provides  a  common  platform  from  which  all  func¬ 
tional  disciplines  (design,  test,  manufacturing,  logistics,  training,  operations,  etc.)  can  work. 


Virtual  Reality:  1.  Also  referred  to  as  artificial  reality  or  synthetic  reality.  Virtual  reality 
perceives  a  participant’s  action  in  terms  of  the  body’s  relationship  to  a  graphic  world  and 
generates  responses  that  maintain  the  illusion  that  his  actions  are  taking  place  within  that 
world.  (Krueger)  2.  The  application  of  integrated  technologies  to  enable  a  participant  to 
sense  that  he  or  she  is  occupying,  to  some  degree,  an  environment  other  than  which  he  or 
she  physically  occupies.  (Loftin) 

Validation:  The  process  of  determining  (a)  the  manner  and  degree  to  which  a  model  is  an 
accurate  representation  of  the  real-world  from  the  perspective  of  the  intended  uses  of  the 
model,  and  (b)  the  confidence  that  should  be  placed  on  this  assessment.  (DoDD  5000.59) 

Verification:  The  process  of  determining  that  a  model  implementation  accurately  repre¬ 
sents  the  developer’s  conceptual  description  and  specifications.  (DoDD  5000.59) 

Weapons  and  Tactics  Center  (WEPTAC):  WEPTAC  is  an  interactive  wargaming  model 
used  for  many-on-may,  multi-platform  scenarios.  It  is  used  for  development  of  fleet  tactics, 
as  well  as  evaluating  the  military  worth  of  future  weapon  systems.  (Navy  M&S  Catalog) 
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